OpenObserve On-Premise Deployment Issue and Resolution

Andrew

Hey all, I'm demoing OpenObserve on-premise, deployed in my k8s cluster (v1.28.1) using Helm (chart version 0.6.4). App is deployed behind a nginx ingress. Setup seems fine, pods are all up and healthy, but when I try to use the app, I get 500 errors in the web console. For now, focusing on logs, when I try to query logs I get a 503 error from `` with a response of `No online querier nodes`. My querier pod is up. I'll add some more details in the thread, with some log snippets

Andrew

Router logs, RUST_LOG is set to debug

Andrew

Prabhat

Looks like somehow querier is not connected to etcd

Prabhat

how are etcd pods doing?

Andrew

healthy

Andrew

Pods are up, no errors in the logs

Prabhat

I see logs logs from fluentbit. So UI and ingestion is working fine. Only querying fails. right?

Andrew

Yup, the UI is just blank

Andrew

Prabhat

can we have a quick call? I can DM you.

Andrew

Sure

Andrew

We fixed this in google meet. Discussed possibly a race condition. Restarted the router, then restarted the querier

Andrew

Logs from fluentbit and metrics from the otel collector are flowing in quite nicely. That's about 9 hours of logs. I'm impressed by the compression to disk :slightly_smiling_face:

Prabhat

:tada: :rocket:

Prabhat

Since you are using both otwl-collwctor and fluentbit, would you want to give this OpenObserve collector helm chart. It is basically otel collector that helps you gather both logs and metrics

Andrew

I did check out that helm chart, but in the end used the OTEL operator to set it all up. I've already had OTEL running, shifting it to OpenObserve was as easy as adding the right endpoints Fluentbit is collecting the logs from my nodes and physical infrastructure over syslog and TCP json and adding it into Openobserve as well. This is something I wasn't able to accomplish with groundcover :smile:

Prabhat

BTW, helm chart leverages OTEL operator, It is configured to allow for capturing logs, metrics and traces (via automatic instrumentation) for nodejs, python, java, .Net and Go.