TLDR Hakan is facing problem with Kubernetes label ingestion into OpenObserve from different services. Hengfei suggested several possibilities including, the use of 'gzip', and verified the absence of data filtration in OpenObserve. The bug remains unresolved.
i am not sure why only some labels ingested, but i am thinking we don't support `gzip` compress, change it to `none` ?
thanks for the suggestion, I changed that, although it looks like it did not block logs from getting ingested before and didn't have any impact on the described behaviour
thanks, i will test with `compress:gzip`
but there is no reason, only some labels ingested, we accept data by row, accept a row or drop a row, and never drop a label.
did you add some function for the stream?
nope, none so far
although I was just thinking about how to eliminate all the keys with `null` values
can you please share an sample log record
and also mention which all labels are diffrenet
```{
"_p": "F",
"_timestamp": 1694519849343055,
"kubernetes_annotations_timestamp": "20230905074108",
"kubernetes_container_hash": "
on this cluster, the logs of the first service get ingested, but not of the second service:
```service-example/component: microservice-example-core
however, on another cluster, the second service does get ingested, with exactly the same label set
OpenObserve has no filter for logs. i am thinking if the fluent-bit deployed to each node?
yes, made sure every node has a running fluent-bit deployed, especially the node with the not-logged service
just wanted to add that on the second cluster, where the second application logs from the above example get ingested, there are other application that again do not get ingested. seems like there isn't an obvious pattern here
i am thinking maybe we can debug for fluent-bit like print the logs instead ingest to OpenObserve to confirm if it collects the logs of that service.
will try that
Hakan
Tue, 12 Sep 2023 11:21:30 UTCHi, I don't understand why only some of the service logs (or their labels) from different clusters are making it into openobserve. Mainly, I am interested in filtering by `kubernetes_labels_app_kubernetes_io_name` which translates to the Kubernetes label `` but somehow, I only see a handful of the resources with that label showing up.
I have this in fluent-bit configured:
``` [OUTPUT]
Name http
Match *
URI /api/test/test-a1/_json
Host
Port 443
tls On
Format json
Json_date_key _timestamp
Json_date_format iso8601
HTTP_User
HTTP_Passwd Complexpass#123
compress gzip```
Any ideas? Thanks for your input.