Trouble Running OpenObserve with Docker and GCS Storage

TLDR Jay had trouble running openobserve with Docker and GCS. Prabhat suggested adding different environments. After resolving this, Jay still had data storage issues, which were clarified by Hengfei.

Photo of Jay
Jay
Tue, 29 Aug 2023 17:54:17 UTC

Hi Team, I'm having trouble running openobserve with docker and using gcs as storage, here is the command ```docker run -p 5080:5080 \ -e RUST_BACKTRACE=1 \ -e ZO_ROOT_USER_EMAIL="" \ -e ZO_ROOT_USER_PASSWORD="Hello@1234" \ -e ZO_LOCAL_MODE_STORAGE="s3" \ -e ZO_S3_SERVER_URL="" \ -e ZO_S3_BUCKET_NAME="<bucker_name>" \ -e ZO_S3_REGION_NAME="auto" \ -e ZO_S3_ACCESS_KEY="<access_key>" \ -e ZO_S3_SECRET_KEY="<secret_key>" \ -e ZO_S3_PROVIDER="gcs" \ public.ecr.aws/zinclabs/openobserve:latest``` and I got an error below looks below ```[2023-08-29T17:53:52Z INFO openobserve::service::db::alerts::destinations] Alert destinations Cached [2023-08-29T17:53:52Z INFO openobserve::service::db::syslog] SyslogRoutes Cached [2023-08-29T17:53:52Z INFO openobserve::service::db::syslog] SyslogServer settings Cached thread 'main' panicked at 'gcp init config error: Generic { store: "GCS", source: Credential { source: OpenCredentials { source: Os { code: 2, kind: NotFound, message: "No such file or directory" }, path: "GOOG1EJFNV6MCUE5Z2EORKKI7VDEF4QRLJHBIQYRMXL3G4PJFTOLN4KLCK4YR" } } }', src/common/infra/storage/remote.rs:293:17``` Can someone pls help me on this? Thank you

Photo of Prabhat
Prabhat
Tue, 29 Aug 2023 19:04:34 UTC

Can you add/replace these to env: ```-e ZO_S3_FEATURE_HTTP1_ONLY="true" \ -e ZO_S3_PROVIDER="s3" \```

Photo of Jay
Jay
Wed, 30 Aug 2023 07:19:31 UTC

Thanks for the solution; it no longer shows an error. However, I've noticed a difference between running the application with the binary file and running it with Docker in terms of data storage: • When using the binary file, the data is stored successfully in Google Cloud Storage (GCS) • However, when using Docker, the data is not being stored in GCS. Here are the commands With docker ```docker run --name openobserve -p 5080:5080 \ -e RUST_BACKTRACE="1" \ -e ZO_ROOT_USER_EMAIL="" \ -e ZO_ROOT_USER_PASSWORD="Hello#1234" \ -e ZO_LOCAL_MODE_STORAGE="s3" \ -e ZO_S3_SERVER_URL="" \ -e ZO_S3_BUCKET_NAME="my_bucket" \ -e ZO_S3_REGION_NAME="asia-southeast1" \ -e ZO_S3_ACCESS_KEY="<access>" \ -e ZO_S3_SECRET_KEY="<secret>" \ -e ZO_S3_FEATURE_HTTP1_ONLY="true" \ -e ZO_S3_PROVIDER="s3" \ public.ecr.aws/zinclabs/openobserve:v0.5.2 ``` ``` With binary file ```RUST_BACKTRACE="1" \ ZO_ROOT_USER_EMAIL="" \ ZO_ROOT_USER_PASSWORD="Hello#1234" \ ZO_LOCAL_MODE_STORAGE="s3" \ ZO_S3_SERVER_URL="" \ ZO_S3_BUCKET_NAME="my_bucket" \ ZO_S3_REGION_NAME="asia-southeast1" \ ZO_S3_ACCESS_KEY="<access>" \ ZO_S3_SECRET_KEY="<secret>" \ ZO_S3_FEATURE_HTTP1_ONLY="true" \ ZO_S3_PROVIDER="s3" \ ./openobserve``` If you found incorrectness on the commands, please let me know. Thank you :pray::skin-tone-2:

Photo of Hengfei
Hengfei
Wed, 30 Aug 2023 07:39:52 UTC

nothing special, we have WAL in node, you need to wait 10 minutes , it will upload to s3

Photo of Jay
Jay
Thu, 31 Aug 2023 08:51:26 UTC

Oh, I got it, How long will it take to remove the data in the local disk after uploading to gcs or s3?

Photo of Hengfei
Hengfei
Thu, 31 Aug 2023 08:55:13 UTC

Photo of Jay
Jay
Thu, 31 Aug 2023 09:46:25 UTC

Very helpful, thank you..

Photo of Jay
Jay
Fri, 01 Sep 2023 03:57:22 UTC

In the document, there is `Ogrnazations` menu But I didn't see it, in the dashboard. did I miss some configurations? Thank you :pray:

Photo of Hengfei
Hengfei
Fri, 01 Sep 2023 04:15:19 UTC

Photo of Hengfei
Hengfei
Fri, 01 Sep 2023 04:15:32 UTC

open source version, have no organization management.