Troubleshooting Openobserve Issue On Linux Machine
TLDR Sushma encountered difficulties viewing pods and creating multiple streams in Openobserve. Prabhat and Hengfei suggested upgrading the software, adjusting SQL syntax, and amending the configuration setup. Despite progress, the issue remains partially unresolved.
2
Nov 14, 2023 (2 weeks ago)
Sushma
06:44 AMI setup the self hosted openobserve on a linux machine, and configured the fluent bit as a daemonset to collect and push logs from minikube deployments to self hosted openobserve.
But I am unable to see all the pods which are running in my minikube.
And please help me with proper SQL syntax to check for specific pod logs.
Openobserve Version : v0.6.4
Let me know if you need any other information.
Prabhat
03:15 PMPrabhat
03:15 PMPrabhat
03:17 PMNov 15, 2023 (2 weeks ago)
Sushma
07:40 AMI would like to create multiple streams in openobserve.
Apart from default, able to create a new stream.
But when I am trying to create another new stream, changes aren't reflecting.
Please help on this..
Hengfei
07:41 AMSushma
07:52 AMI have defined the output section like this in fliebeat configmap.
[OUTPUT]
URI /api/default/allfunds/_json
Host **
Port 5080
tls Off
Format json
Json_date_key _timestamp
Json_date_format iso8601
HTTP_User *
HTTP_Passwd *
compress gzip
Here. "allfunds" is my stream name
Hengfei
07:54 AMHengfei
07:54 AMHengfei
07:54 AMSushma
08:54 AMSushma
09:17 AMBut unable to see any pods in openobserve UI, checked the fluent-bit logs, getting the below warning..
[2023/11/15 09:14:55] [ warn] [engine] failed to flush chunk '1-1700036229.123663863.flb', retry in 1508 seconds: task_id=1751, input=tail.1 > output=es.0 (out_id=0)
[2023/11/15 09:14:56] [ warn] [engine] failed to flush chunk '1-1700035046.129490961.flb', retry in 14 seconds: task_id=565, input=tail.1 > output=es.0 (out_id=0)
[2023/11/15 09:14:56] [ warn] [engine] failed to flush chunk '1-1700034792.129911216.flb', retry in 947 seconds: task_id=311, input=tail.1 > output=es.0 (out_id=0)
Please help me to resolve this..
Sushma
09:19 AMCurerntly using the openobserve version 0.7.0
Hengfei
09:20 AMHengfei
09:20 AMHengfei
09:21 AMHengfei
09:21 AMSushma
09:28 AMFor now , I have 2 streams..
->Allfunds
->Kestra
In the UI, I am unable to fetch the pod names for allfunds, openobserve is throwing an error like "Unable to fetch the field values "
Please find the screenshot for reference.
Sushma
09:29 AM[2023/11/15 07:48:17] [ info] [output:http:http.2] 172.16.22.22:5080, HTTP status=200
{"code":200,"status":[{"name":"kestra","successful":4,"failed":0}]}
[2023/11/15 07:48:18] [ warn] [engine] failed to flush chunk '1-1700034497.124087455.flb', retry in 8 seconds: task_id=16, input=tail.1 > output=es.0 (out_id=0)
[2023/11/15 07:48:18] [ warn] [engine] failed to flush chunk '1-1700034491.123633763.flb', retry in 18 seconds: task_id=10, input=tail.1 > output=es.0 (out_id=0)
Hengfei
09:29 AM[2023/11/15 07:48:18] [ warn] [engine] failed to flush chunk '1-1700034491.123633763.flb', retry in 18 seconds: task_id=10, input=tail.1 > output=es.0 (out_id=0)
Hengfei
09:29 AMes.0
output has problem.Hengfei
09:30 AMSushma
09:30 AMSushma
09:34 AM[PARSER]
Name dockerno_time
Format json
Time_Keep Off
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
fluent-bit.conf: |
[SERVICE]
Daemon Off
Flush 1
Log_Level info
Parsers_File /fluent-bit/etc/parsers.conf
Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
Health_Check On
[INPUT]
Name tail
Path /var/log/containers/*_allfunds.log
multiline.parser docker, cri
Tag kube.
MemBuf_Limit 5MB
Skip_Long_Lines On
[INPUT]
Name tail
Path /var/log/containers/*_kestra.log
multiline.parser docker, cri
Tag kube.
MemBuf_Limit 5MB
Skip_Long_Lines On
[INPUT]
Name systemd
Tag host.*
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
Read_From_Tail On
[OUTPUT]
Name es
Match *
Host *
Logstash_Format On
Retry_Limit False
[OUTPUT]
Name http
Match *_allfunds*
URI /api/default/allfunds/json
Host *
Port 5080
tls Off
Format json
Json_date_key _timestamp
Json_date_format iso8601
HTTP_User **
HTTP_Passwd *
compress gzip
[OUTPUT]
Name http
Match *_kestra*
URI /api/default/kestra/_json
Host **
Port 5080
tls Off
Format json
Json_date_key _timestamp
Json_date_format iso8601
HTTP_User **
HTTP_Passwd **
compress gzip
Hengfei
09:35 AM[OUTPUT]
Name es
Match *
Host ***
Logstash_Format On
Retry_Limit False
Hengfei
09:35 AMHengfei
09:36 AMSushma
09:36 AMHengfei
09:37 AMHengfei
09:37 AMHengfei
09:37 AMHengfei
09:40 AMSushma
09:46 AMBut in Openobserve UI, unable to see the pod names..
kubernetes_pod_name field is showing null..
Sushma
09:46 AMSushma
09:55 AMHengfei
10:02 AMHengfei
10:05 AMapiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit
namespace: logging
labels:
: fluent-bit
: Helm
: fluent-bit
: 2.0.8
: fluent-bit-0.21.7
annotations:
: fluent-bit
: logging
data:
custom_parsers.conf: |
[PARSER]
Name docker_no_time
Format json
Time_Keep Off
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
fluent-bit.conf: |
[SERVICE]
Daemon Off
Flush 1
Log_Level info
Parsers_File parsers.conf
Parsers_File custom_parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
Health_Check On
[INPUT]
Name tail
Path /var/log/containers/*.log
multiline.parser docker, cri
Tag kube.*
Mem_Buf_Limit 5MB
Skip_Long_Lines On
[INPUT]
Name systemd
Tag host.*
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
Read_From_Tail On
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Keep_Log Off
K8S-Logging.Parser On
K8S-Logging.Exclude On
# [OUTPUT]
# Name es
# Match kube.*
# Host elasticsearch-master
# Logstash_Format On
# Retry_Limit False
[OUTPUT]
Name http
Match *
URI /api/default/alpha2/_json
Host zinclabs.dev
Port 443
tls On
Format json
Json_date_key _timestamp
Json_date_format iso8601
HTTP_User
HTTP_Passwd *****
compress gzip
Hengfei
10:05 AMfilter
but you don't have it.Hengfei
10:05 AMSushma
10:06 AMSushma
10:15 AMIn my "allfunds" namespace, having around 10 pods, but in Openobserve UI , only one pod is visible, and remaining all are invisble..
Hengfei
10:17 AMHengfei
10:17 AMHengfei
10:17 AMHengfei
10:18 AMSushma
10:18 AMEarlier we used to get unable to fetch the field values for all fields..
Sushma
10:19 AM1
Sushma
10:35 AM1
Hengfei
10:37 AMHengfei
10:37 AMOpenObserve
Indexed 406 threads (74% resolved)
Similar Threads
Testing Openobserve: Fluent-bit Config & Network Errors.
vasanth needed help testing Openobserve. Ashish provided information for fluent-bit configuration and addressed network error questions.
Trouble Running OpenObserve with Docker and GCS Storage
Jay had trouble running openobserve with Docker and GCS. Prabhat suggested adding different environments. After resolving this, Jay still had data storage issues, which were clarified by Hengfei.
OpenObserve Support for Single Sign-On and Self-Hosting
Gabriel inquired about OpenObserve supporting single sign-on, but Prabhat clarified it wasn't available for self-hosted versions yet and suggested creating new orgs as a workaround for managing access. Karan agreed to monitor the roadmap for future updates.
Troubleshooting openobserve HA Self-hosted Set Up Error
Dhananjay faced an issue querying logs with their openobserve HA setup. Ashish, Prabhat, and Joaquin provided troubleshooting steps. The issue was resolved by increasing the read-timeout.
Fixing openobserve with rust tracing and opentelemetry-otlp issue
Gunnar had problems with openobserve using rust tracing and opentelemetry-otlp. Ashish and West provided solutions to resolve the issue and discussed matter related to gRPC and HTTP.