Troubleshooting Openobserve Issue On Linux Machine

Regarding SQL syntax - Take a look at example queries here - https://openobserve.ai/docs/example-queries/

There are a lot of functions available here - https://openobserve.ai/docs/functions/ . You could use these too

and, you can have a look the logs of openobserve, it shows the http requet logs for default.

You can see what is the ingest request

I am able to create multiple streams now in openobserve,
But unable to see any pods in openobserve UI, checked the fluent-bit logs, getting the below warning..

[2023/11/15 09:14:55] [ warn] [engine] failed to flush chunk '1-1700036229.123663863.flb', retry in 1508 seconds: task_id=1751, input=tail.1 > output=es.0 (out_id=0)
[2023/11/15 09:14:56] [ warn] [engine] failed to flush chunk '1-1700035046.129490961.flb', retry in 14 seconds: task_id=565, input=tail.1 > output=es.0 (out_id=0)
[2023/11/15 09:14:56] [ warn] [engine] failed to flush chunk '1-1700034792.129911216.flb', retry in 947 seconds: task_id=311, input=tail.1 > output=es.0 (out_id=0)

Please help me to resolve this..

I am not getting any "error" logs in openobserve.
Curerntly using the openobserve version 0.7.0

For now, Is it working with the default stream?

maybe you need scrollup and check the root error in fluent-bit logs.

this error tell us nothing.

and below is the full trace of error..

[2023/11/15 07:48:17] [ info] [output:http:http.2] 172.16.22.22:5080, HTTP status=200
{"code":200,"status":[{"name":"kestra","successful":4,"failed":0}]}
[2023/11/15 07:48:18] [ warn] [engine] failed to flush chunk '1-1700034497.124087455.flb', retry in 8 seconds: task_id=16, input=tail.1 > output=es.0 (out_id=0)
[2023/11/15 07:48:18] [ warn] [engine] failed to flush chunk '1-1700034491.123633763.flb', retry in 18 seconds: task_id=10, input=tail.1 > output=es.0 (out_id=0)

it report es.0 output has problem.

what is the first output of es?

This is the complete fluent-bit configuration


[PARSER]
Name dockerno_time
Format json
Time_Keep Off
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
fluent-bit.conf: |
[SERVICE]
Daemon Off
Flush 1
Log_Level info
Parsers_File /fluent-bit/etc/parsers.conf
Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
Health_Check On

[INPUT]
Name tail
Path /var/log/containers/*_allfunds.log
multiline.parser docker, cri
Tag kube.
MemBuf_Limit 5MB
Skip_Long_Lines On

[INPUT]
Name tail
Path /var/log/containers/*_kestra.log
multiline.parser docker, cri
Tag kube.
MemBuf_Limit 5MB
Skip_Long_Lines On

[INPUT]
Name systemd
Tag host.*
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
Read_From_Tail On

[OUTPUT]
Name es
Match *
Host *
Logstash_Format On
Retry_Limit False

[OUTPUT]
Name http
Match *_allfunds*
URI /api/default/allfunds/json
Host *
Port 5080
tls Off
Format json
Json_date_key _timestamp
Json_date_format iso8601
HTTP_User **
HTTP_Passwd *
compress gzip

[OUTPUT]
Name http
Match *_kestra*
URI /api/default/kestra/_json
Host **
Port 5080
tls Off
Format json
Json_date_key _timestamp
Json_date_format iso8601
HTTP_User **
HTTP_Passwd **
compress gzip

this is the es.0

from the logs, it reports es.0, has some problem.

it should report something in the logs in earlier.

you should check all of the fluentbit logs.

just recreate the fluentbit, then check the logs, it should report the error at beginning.

Logs are dumping to openobserve, but podnames are invisible

Not only "kubernetes_pod_name", every other filed is throwing the same error like "unable to fetch the field values" in version 0.7.0

This is my configure:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit
  namespace: logging
  labels:
    : fluent-bit
    : Helm
    : fluent-bit
    : 2.0.8
    : fluent-bit-0.21.7
  annotations:
    : fluent-bit
    : logging
data:
  custom_parsers.conf: |
    [PARSER]
        Name docker_no_time
        Format json
        Time_Keep Off
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
  fluent-bit.conf: |
    [SERVICE]
        Daemon Off
        Flush 1
        Log_Level info
        Parsers_File parsers.conf
        Parsers_File custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020
        Health_Check On

    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser docker, cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

    [INPUT]
        Name systemd
        Tag host.*
        Systemd_Filter _SYSTEMD_UNIT=kubelet.service
        Read_From_Tail On

    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

    # [OUTPUT]
    #     Name es
    #     Match kube.*
    #     Host elasticsearch-master
    #     Logstash_Format On
    #     Retry_Limit False
  
    [OUTPUT]
        Name http
        Match *
        URI /api/default/alpha2/_json
        Host zinclabs.dev
        Port 443
        tls On
        Format json
        Json_date_key    _timestamp
        Json_date_format iso8601
        HTTP_User 
        HTTP_Passwd *****
        compress gzip

i saw i have filter but you don't have it.

can you try add the filter section?

I have added the FILTER section..
In my "allfunds" namespace, having around 10 pods, but in Openobserve UI , only one pod is visible, and remaining all are invisble..

you need to know that is about ingest part have no the labels.

after added the filter, we already have one pod name, right?

let's wait for 10 minutes, let more data ingested, let's see, if it report more pod name.

Sure, lets wait for sometime and see

Now, the kubernetes_pod_names field is displaying null value and along with other pod names..

you can check what is the logs.