Troubleshooting Openobserve Setup with Azure Blob on K8S

TLDR David had difficulties setting up Openobserve with Azure Blob in Kubernetes. Prabhat provided guidance and setup values, informing David about possible data transmission delays, which resolved the issue.

Photo of David
David
Fri, 13 Oct 2023 12:51:57 UTC

Hi I am trying to use Openobserve in HA with Azure blob on K8S. Which values should I use for • ZO_S3_ACCESS_KEY • ZO_S3_SERVER_URL: Side note. It is possible to use a secret to inject the `ZO_S3_SECRET_KEY` Thanks

Photo of Prabhat
Prabhat
Fri, 13 Oct 2023 13:55:01 UTC

Hi David, While the library that we use supports Azure blob, we have not tested it ourselves yet. I will be testing it today but if you want to give it a shot ahead of me, you can try setting the env variables at this link

Photo of David
David
Fri, 13 Oct 2023 13:57:16 UTC

thanks Prabhat

Photo of Prabhat
Prabhat
Fri, 13 Oct 2023 21:52:19 UTC

David You can use the settings here -

Photo of David
David
Wed, 25 Oct 2023 13:58:15 UTC

Prabhat Hi. I am trying to set up openobserve with ``ZO_LOCAL_MODE_STORAGE=s3`` on azure with the updated documentation that you send me. I am actually trying this time locally on docker. I can see the data on the openobserve logs page but I do not see the data on azure. I cannot see anything in the logs related to azure in general. Could you please help thanks

Photo of David
David
Wed, 25 Oct 2023 14:04:49 UTC

`[2023-10-25T14:02:21Z INFO openobserve] Starting OpenObserve v0.6.4 [2023-10-25T14:02:21Z INFO openobserve] System info: CPU cores 20, MEM total 31758 MB, Disk total 464 GB, free 308 GB [2023-10-25T14:02:22Z INFO openobserve::common::migration] Upgrading from v0.0.0 to v0.6.4 [2023-10-25T14:02:22Z INFO openobserve::service::db::user] Start watching user [2023-10-25T14:02:22Z INFO openobserve::service::db::user] Users Cached [2023-10-25T14:02:22Z INFO openobserve::service::db::compact::retention] Start watching stream deleting [2023-10-25T14:02:22Z INFO openobserve::service::db::schema] Stream schemas Cached [2023-10-25T14:02:22Z INFO openobserve::service::db::metrics] Start watching prometheus cluster leader [2023-10-25T14:02:22Z INFO openobserve::service::db::schema] Start watching stream schema [2023-10-25T14:02:22Z INFO openobserve::service::db::functions] Start watching function [2023-10-25T14:02:22Z INFO openobserve::service::db::alerts::templates] Start watching alert templates [2023-10-25T14:02:22Z INFO openobserve::service::db::alerts::destinations] Start watching alert destinations [2023-10-25T14:02:22Z INFO openobserve::service::db::triggers] Start watching Triggers [2023-10-25T14:02:22Z INFO openobserve::service::db::alerts] Start watching alerts [2023-10-25T14:02:22Z INFO openobserve::service::db::functions] Functions Cached [2023-10-25T14:02:22Z INFO openobserve::service::db::metrics] Prometheus cluster leaders Cached [2023-10-25T14:02:22Z INFO openobserve::service::db::alerts::templates] Alert templates Cached [2023-10-25T14:02:22Z INFO openobserve::service::db::alerts::destinations] Alert destinations Cached [2023-10-25T14:02:22Z INFO openobserve::service::db::alerts] Alerts Cached [2023-10-25T14:02:22Z INFO openobserve::service::db::triggers] Triggers Cached [2023-10-25T14:02:22Z INFO openobserve::service::db::syslog] SyslogRoutes Cached [2023-10-25T14:02:22Z INFO openobserve::service::db::syslog] SyslogServer settings Cached [2023-10-25T14:02:22Z INFO openobserve::job] Job initialization complete [2023-10-25T14:02:22Z INFO openobserve::service::db::schema] EnrichmentTables Cached [2023-10-25T14:02:22Z INFO openobserve::service::db::syslog] Start watching SyslogServer settings [2023-10-25T14:02:22Z INFO openobserve] starting gRPC server at 0.0.0.0:5081 [2023-10-25T14:02:22Z INFO openobserve::service::db::syslog] Start watching syslog routes [2023-10-25T14:02:22Z INFO openobserve::common::meta::telemetry] sending event OpenObserve - Starting server [2023-10-25T14:02:23Z INFO actix_server::builder] starting 20 workers [2023-10-25T14:02:23Z INFO actix_server::server] Tokio runtime found; starting in existing Tokio runtime [2023-10-25T14:02:23Z INFO openobserve] starting HTTP server at: 0.0.0.0:5080, thread_id: 0 [2023-10-25T14:02:23Z INFO openobserve] starting HTTP server at: 0.0.0.0:5080, thread_id: 0 [2023-10-25T14:02:23Z INFO openobserve] starting HTTP server at: 0.0.0.0:5080, thread_id: 0 [2023-10-25T14:02:23Z INFO openobserve] starting HTTP server at: 0.0.0.0:5080, thread_id: 0 [2023-10-25T14:02:23Z INFO openobserve] starting HTTP server at: 0.0.0.0:5080, thread_id: 0 [2023-10-25T14:02:23Z INFO openobserve] starting HTTP server at: 0.0.0.0:5080, thread_id: 0 [2023-10-25T14:02:23Z INFO openobserve] starting HTTP server at: 0.0.0.0:5080, thread_id: 0 [2023-10-25T14:02:23Z INFO openobserve] starting HTTP server at: 0.0.0.0:5080, thread_id: 0 [2023-10-25T14:02:23Z INFO openobserve] starting HTTP server at: 0.0.0.0:5080, thread_id: 0 [2023-10-25T14:02:23Z INFO openobserve] starting HTTP server at: 0.0.0.0:5080, thread_id: 0 [2023-10-25T14:02:23Z INFO openobserve] starting HTTP server at: 0.0.0.0:5080, thread_id: 0 [2023-10-25T14:02:23Z INFO openobserve] starting HTTP server at: 0.0.0.0:5080, thread_id: 0 [2023-10-25T14:02:23Z INFO openobserve] starting HTTP server at: 0.0.0.0:5080, thread_id: 0 [2023-10-25T14:02:23Z INFO openobserve] starting HTTP server at: 0.0.0.0:5080, thread_id: 0 [2023-10-25T14:02:23Z INFO openobserve] starting HTTP server at: 0.0.0.0:5080, thread_id: 0 [2023-10-25T14:02:23Z INFO openobserve] starting HTTP server at: 0.0.0.0:5080, thread_id: 0 [2023-10-25T14:02:23Z INFO openobserve] starting HTTP server at: 0.0.0.0:5080, thread_id: 0 [2023-10-25T14:02:23Z INFO openobserve] starting HTTP server at: 0.0.0.0:5080, thread_id: 0 [2023-10-25T14:02:23Z INFO openobserve] starting HTTP server at: 0.0.0.0:5080, thread_id: 0 [2023-10-25T14:02:23Z INFO openobserve] starting HTTP server at: 0.0.0.0:5080, thread_id: 0 [2023-10-25T14:02:23Z INFO openobserve::service::schema] Acquired lock for stream vector as schema is empty [2023-10-25T14:02:23Z INFO openobserve::service::schema] Setting schema for stream vector as schema is empty` this is the only thing that I see apart some http post 200 from vector my collector agent

Photo of Prabhat
Prabhat
Wed, 25 Oct 2023 17:56:33 UTC

OpenObserve keeps the data locally and batches it before pushing it to object store. So you might see a delay (upto 10 minutes) based on the volume of data that you push to OpenObserve.

Photo of David
David
Thu, 26 Oct 2023 08:54:48 UTC

Thanks Prabhat, will I see any logs that openobserve is trying to send batches to object store?

Photo of Prabhat
Prabhat
Thu, 26 Oct 2023 10:28:05 UTC

Yes, you should see logs in OpenObserve. Something like ```[2023-10-25T22:44:29Z INFO openobserve::job::files::disk] [JOB] File upload begin: disk: ./data/wal/files/default/logs/_sessionreplay/0/2023/10/25/22/56f9ba8bfd0dfff3/7123074350588231680xle5bW.json [2023-10-25T22:44:29Z INFO openobserve::job::files::disk] [JOB] disk file upload succeeded: files/myorg/logs/_rumlog/2023/10/25/22/7123076884954480640qo0hjQ.parquet```

Photo of David
David
Sun, 05 Nov 2023 14:36:34 UTC

It worked. Thanks

OpenObserve Logo

OpenObserve

Indexed 419 threads

OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI

View in App

Similar Threads

Kubernetes Deployment Issue and Resolution with Postgres and Azure Blob
Integrating OpenObserve with a User Solution and Handling Errors.
Troubleshooting openobserve HA Self-hosted Set Up Error
OpenObserve Errors and How to Fix Them
OpenObserve Post Request Issues and Discussion on Logging