Issues with Partial Kubernetes Label Ingestion into OpenObserve
TLDR Hakan is facing problem with Kubernetes label ingestion into OpenObserve from different services. Hengfei suggested several possibilities including, the use of 'gzip', and verified the absence of data filtration in OpenObserve. The bug remains unresolved.
Hengfei
Tue, 12 Sep 2023 11:23:42 UTCi am not sure why only some labels ingested, but i am thinking we don't support `gzip` compress, change it to `none` ?
Hakan
Tue, 12 Sep 2023 11:28:54 UTCthanks for the suggestion, I changed that, although it looks like it did not block logs from getting ingested before and didn't have any impact on the described behaviour
Hengfei
Tue, 12 Sep 2023 11:45:46 UTCthanks, i will test with `compress:gzip`
Hengfei
Tue, 12 Sep 2023 11:46:25 UTCbut there is no reason, only some labels ingested, we accept data by row, accept a row or drop a row, and never drop a label.
Hengfei
Tue, 12 Sep 2023 11:46:50 UTCdid you add some function for the stream?
Hakan
Tue, 12 Sep 2023 11:47:14 UTCnope, none so far
Hakan
Tue, 12 Sep 2023 11:48:23 UTCalthough I was just thinking about how to eliminate all the keys with `null` values
Ashish
Tue, 12 Sep 2023 11:55:00 UTCcan you please share an sample log record
Ashish
Tue, 12 Sep 2023 11:55:22 UTCand also mention which all labels are diffrenet
Hakan
Tue, 12 Sep 2023 11:58:56 UTC```{
"_p": "F",
"_timestamp": 1694519849343055,
"kubernetes_annotations_timestamp": "20230905074108",
"kubernetes_container_hash": "
Hakan
Tue, 12 Sep 2023 12:02:17 UTCon this cluster, the logs of the first service get ingested, but not of the second service:
```service-example/component: microservice-example-core
Hakan
Tue, 12 Sep 2023 12:04:19 UTChowever, on another cluster, the second service does get ingested, with exactly the same label set
Hengfei
Tue, 12 Sep 2023 12:05:18 UTCOpenObserve has no filter for logs. i am thinking if the fluent-bit deployed to each node?
Hakan
Tue, 12 Sep 2023 12:08:03 UTCyes, made sure every node has a running fluent-bit deployed, especially the node with the not-logged service
Hakan
Tue, 12 Sep 2023 12:16:38 UTCjust wanted to add that on the second cluster, where the second application logs from the above example get ingested, there are other application that again do not get ingested. seems like there isn't an obvious pattern here
Hengfei
Tue, 12 Sep 2023 12:18:36 UTCi am thinking maybe we can debug for fluent-bit like print the logs instead ingest to OpenObserve to confirm if it collects the logs of that service.
Hakan
Tue, 12 Sep 2023 12:21:47 UTCwill try that
OpenObserve
Indexed 419 threads
OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI
View in App
Hakan
Tue, 12 Sep 2023 11:21:30 UTCHi, I don't understand why only some of the service logs (or their labels) from different clusters are making it into openobserve. Mainly, I am interested in filtering by `kubernetes_labels_app_kubernetes_io_name` which translates to the Kubernetes label `` but somehow, I only see a handful of the resources with that label showing up.
I have this in fluent-bit configured:
``` [OUTPUT]
Name http
Match *
URI /api/test/test-a1/_json
Host
Port 443
tls On
Format json
Json_date_key _timestamp
Json_date_format iso8601
HTTP_User
HTTP_Passwd Complexpass#123
compress gzip```
Any ideas? Thanks for your input.