Integrating ZincObserver and Resolving Log Issues

TLDR sparrow faced issues with ZincObserver integration, log filtering, and missing logs. Ashish and Hengfei provided solutions, including using str_match and an updated image. A future release will provide more error responses for ingestion.

Photo of sparrow
sparrow
Fri, 31 Mar 2023 10:58:09 UTC

Hello Team, I was trying to integrate ZincObserver to our cluster, and it’s deployed successfully. But have few questions. 1. i’m not able to find the logs accordingly 2. Wants to send the data to s3 and fetch the data from s3 only 3. Want to apply delete policy for specific time interval How to achieve all these things?

Photo of Ashish
Ashish
Fri, 31 Mar 2023 11:00:36 UTC

1. Can you elaborate what do you mean by not able to find logs?

Photo of Ashish
Ashish
Fri, 31 Mar 2023 11:02:29 UTC

2. to send data to s3 , if you are using local mode set ZO_LOCAL_MODE_STORAGE = “s3” in env variables

Photo of sparrow
sparrow
Fri, 31 Mar 2023 11:02:34 UTC

Actully I’m trying to filter out logs based on some container

Photo of Ashish
Ashish
Fri, 31 Mar 2023 11:02:38 UTC

check this

Photo of Ashish
Ashish
Fri, 31 Mar 2023 11:02:50 UTC

or you can setup cluster mode

Photo of Ashish
Ashish
Fri, 31 Mar 2023 11:03:07 UTC

3. delete policy isnt available yet

Photo of Ashish
Ashish
Fri, 31 Mar 2023 11:03:15 UTC

is on roadmap

Photo of Ashish
Ashish
Fri, 31 Mar 2023 11:03:36 UTC

for search request you to check :

Photo of Ashish
Ashish
Fri, 31 Mar 2023 11:03:45 UTC

if you still face issue

Photo of Ashish
Ashish
Fri, 31 Mar 2023 11:03:55 UTC

let us know specifics of it

Photo of Ashish
Ashish
Fri, 31 Mar 2023 11:04:05 UTC

we will be able to assist

Photo of sparrow
sparrow
Fri, 31 Mar 2023 11:06:58 UTC

I’m not able to find logs for this specific container? Am i doing anything wrong here?

Photo of Ashish
Ashish
Fri, 31 Mar 2023 11:07:44 UTC

• match_all searches only the fields that are configured for full text search. Default set of fields are `msg, message, log, logs`. If you want more fields to be scanned during full text search, you can configure them under stream settings. You should use `str_match` for full text search in specific fields.

Photo of Ashish
Ashish
Fri, 31 Mar 2023 11:08:45 UTC

`try using str_match`

Photo of sparrow
sparrow
Fri, 31 Mar 2023 11:09:02 UTC

how to fliter logs of specific pod

Photo of sparrow
sparrow
Fri, 31 Mar 2023 11:09:04 UTC

?

Photo of Ashish
Ashish
Fri, 31 Mar 2023 11:09:45 UTC

instead of match_all

Photo of Ashish
Ashish
Fri, 31 Mar 2023 11:09:55 UTC

use function str_match

Photo of sparrow
sparrow
Fri, 31 Mar 2023 11:11:21 UTC

nothing is there with the same field

Photo of Ashish
Ashish
Fri, 31 Mar 2023 11:11:59 UTC

str_match(kubernetes_container_name,‘orchestrator’)

Photo of sparrow
sparrow
Fri, 31 Mar 2023 11:13:38 UTC

Nothing

Photo of Ashish
Ashish
Fri, 31 Mar 2023 11:15:22 UTC

do one thing

Photo of Ashish
Ashish
Fri, 31 Mar 2023 11:15:35 UTC

remove everything from search panel

Photo of Ashish
Ashish
Fri, 31 Mar 2023 11:15:46 UTC

add kubernetes_container_name to seach result

Photo of sparrow
sparrow
Fri, 31 Mar 2023 11:15:57 UTC

done now not able to list the index

Photo of Ashish
Ashish
Fri, 31 Mar 2023 11:16:32 UTC

call?

Photo of sparrow
sparrow
Fri, 31 Mar 2023 11:16:52 UTC

yes that would be helpful

Photo of Ashish
Ashish
Fri, 31 Mar 2023 11:17:23 UTC

join huddle

Photo of sparrow
sparrow
Fri, 31 Mar 2023 11:23:32 UTC

{“code”:20008,“message”:“Search SQL execute error”,“error_detail”:“Error during planning: Projection references non-aggregate values: Expression tbl._timestamp could not be resolved from available columns: tbl.key, COUNT(UInt8(1))“}

Photo of Ashish
Ashish
Fri, 31 Mar 2023 11:38:16 UTC

Can you please share backend logs as well

Photo of Ashish
Ashish
Fri, 31 Mar 2023 11:38:53 UTC

I mean ZincObserve server logs

Photo of sparrow
sparrow
Fri, 31 Mar 2023 11:41:34 UTC

[2023-03-31T11:38:56Z INFO zincobserve::service::search::datafusion::exec] Query agg:_count took 2.464 seconds. [2023-03-31T11:38:56Z INFO zincobserve::service::search::datafusion::exec] Query agg sql: select date_bin(interval ‘5 second’, to_timestamp_micros(“_timestamp”), to_timestamp(‘2001-01-01T00:00:00’)) AS key, count(*) AS num FROM tbl WHERE ((_timestamp >= 1680261833538000 AND _timestamp < 1680262733538000) ) GROUP BY key ORDER BY key [2023-03-31T11:38:56Z INFO actix_web::middleware::logger] 10.244.31.9 “POST /api/default/default/_json HTTP/1.1" 200 69 “9208” “-” “Fluent-Bit” 0.005976 [2023-03-31T11:38:56Z INFO actix_web::middleware::logger] 10.244.18.51 “POST /api/default/default/_json HTTP/1.1" 200 68 “1850” “-” “Fluent-Bit” 0.002087 [2023-03-31T11:38:56Z INFO actix_web::middleware::logger] 10.244.18.51 “POST /api/default/default/_json HTTP/1.1" 200 68 “8552” “-” “Fluent-Bit” 0.004857 [2023-03-31T11:38:56Z INFO actix_web::middleware::logger] 10.244.31.9 “POST /api/default/default/_json HTTP/1.1" 200 69 “14241” “-” “Fluent-Bit” 0.029486 [2023-03-31T11:38:56Z ERROR zincobserve::service::search::datafusion::exec] aggs sql execute failed, session: Session { id: “3b970678-1409-4d6f-b5eb-e7c1a395320c”, data_type: Cache }, sql: select date_bin(interval ‘5 second’, to_timestamp_micros(“_timestamp”), to_timestamp(‘2001-01-01T00:00:00’)) AS key, count(*) AS num FROM tbl WHERE ((_timestamp >= 1680261833538000 AND _timestamp < 1680262733538000) ) GROUP BY key ORDER BY key, err: Plan(“Projection references non-aggregate values: Expression tbl._timestamp could not be resolved from available columns: tbl.key, COUNT(UInt8(1))“) [2023-03-31T11:38:56Z ERROR zincobserve::service::search::cache] datafusion execute error: Error during planning: Projection references non-aggregate values: Expression tbl._timestamp could not be resolved from available columns: tbl.key, COUNT(UInt8(1)) [2023-03-31T11:38:56Z ERROR zincobserve::handler::http::request::search] search error: ErrorCode(SearchSQLExecuteError(“Error during planning: Projection references non-aggregate values: Expression tbl._timestamp could not be resolved from available columns: tbl.key, COUNT(UInt8(1))“)) [2023-03-31T11:38:56Z INFO actix_web::middleware::logger] 10.244.31.9 “POST /api/default/_search?type=logs HTTP/1.1” 500 205 “310" “” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 2.622346

Photo of Hengfei
Hengfei
Fri, 31 Mar 2023 11:51:16 UTC

checking

Photo of Hengfei
Hengfei
Fri, 31 Mar 2023 12:05:37 UTC

sorry, i can’t find the problem for this logs, Can you compress the directory `data` and give us for debug?

Photo of Hengfei
Hengfei
Fri, 31 Mar 2023 12:38:35 UTC

get the same error, thanks for you data.

Photo of Hengfei
Hengfei
Fri, 31 Mar 2023 12:38:38 UTC

i will debug for that.

Photo of Hengfei
Hengfei
Fri, 31 Mar 2023 14:10:02 UTC

public.ecr.aws/zinclabs/zincobserve-dev:v0.3.1-75ec795

Photo of Hengfei
Hengfei
Fri, 31 Mar 2023 14:10:18 UTC

sparrow we fixed the issue, you can use this image for test.

Photo of sparrow
sparrow
Mon, 03 Apr 2023 04:45:17 UTC

Thanks Hengfei will check today with this image and let you in any case

Photo of sparrow
sparrow
Mon, 03 Apr 2023 10:22:38 UTC

I thinks it’s working fine now. But have one question, I thinks some logs getting missed from specific pods

Photo of Hengfei
Hengfei
Mon, 03 Apr 2023 10:25:41 UTC

Like what?

Photo of sparrow
sparrow
Mon, 03 Apr 2023 10:26:22 UTC

like in pod logs I can see some error or some details but not visible on zinc

Photo of Ashish
Ashish
Mon, 03 Apr 2023 10:26:52 UTC

do you have any functions

Photo of Ashish
Ashish
Mon, 03 Apr 2023 10:27:12 UTC

on the streams which may result in discarding ingested records?

Photo of sparrow
sparrow
Mon, 03 Apr 2023 10:29:37 UTC

Nope

Photo of Hengfei
Hengfei
Mon, 03 Apr 2023 10:31:11 UTC

Ashish maybe we can let sparrow wait our next release, we will add more error response for ingestion.

Photo of Hengfei
Hengfei
Mon, 03 Apr 2023 10:31:32 UTC

they will see some error if we drop some records.

Photo of sparrow
sparrow
Mon, 03 Apr 2023 10:35:54 UTC

Any ETA for the same?