Integrating ZincObserver and Resolving Log Issues
TLDR sparrow faced issues with ZincObserver integration, log filtering, and missing logs. Ashish and Hengfei provided solutions, including using str_match and an updated image. A future release will provide more error responses for ingestion.
Ashish
Fri, 31 Mar 2023 11:00:36 UTC1. Can you elaborate what do you mean by not able to find logs?
Ashish
Fri, 31 Mar 2023 11:02:29 UTC2. to send data to s3 , if you are using local mode set ZO_LOCAL_MODE_STORAGE = “s3” in env variables
sparrow
Fri, 31 Mar 2023 11:02:34 UTCActully I’m trying to filter out logs based on some container
Ashish
Fri, 31 Mar 2023 11:02:38 UTCcheck this
Ashish
Fri, 31 Mar 2023 11:02:50 UTCor you can setup cluster mode
Ashish
Fri, 31 Mar 2023 11:03:07 UTC3. delete policy isnt available yet
Ashish
Fri, 31 Mar 2023 11:03:15 UTCis on roadmap
Ashish
Fri, 31 Mar 2023 11:03:36 UTCfor search request you to check :
Ashish
Fri, 31 Mar 2023 11:03:45 UTCif you still face issue
Ashish
Fri, 31 Mar 2023 11:03:55 UTClet us know specifics of it
Ashish
Fri, 31 Mar 2023 11:04:05 UTCwe will be able to assist
sparrow
Fri, 31 Mar 2023 11:06:58 UTCI’m not able to find logs for this specific container? Am i doing anything wrong here?
Ashish
Fri, 31 Mar 2023 11:07:44 UTC• match_all searches only the fields that are configured for full text search. Default set of fields are `msg, message, log, logs`. If you want more fields to be scanned during full text search, you can configure them under stream settings. You should use `str_match` for full text search in specific fields.
Ashish
Fri, 31 Mar 2023 11:08:45 UTC`try using str_match`
sparrow
Fri, 31 Mar 2023 11:09:02 UTChow to fliter logs of specific pod
sparrow
Fri, 31 Mar 2023 11:09:04 UTC?
Ashish
Fri, 31 Mar 2023 11:09:45 UTCinstead of match_all
Ashish
Fri, 31 Mar 2023 11:09:55 UTCuse function str_match
sparrow
Fri, 31 Mar 2023 11:11:21 UTCnothing is there with the same field
Ashish
Fri, 31 Mar 2023 11:11:59 UTCstr_match(kubernetes_container_name,‘orchestrator’)
sparrow
Fri, 31 Mar 2023 11:13:38 UTCNothing
Ashish
Fri, 31 Mar 2023 11:15:22 UTCdo one thing
Ashish
Fri, 31 Mar 2023 11:15:35 UTCremove everything from search panel
Ashish
Fri, 31 Mar 2023 11:15:46 UTCadd kubernetes_container_name to seach result
sparrow
Fri, 31 Mar 2023 11:15:57 UTCdone now not able to list the index
Ashish
Fri, 31 Mar 2023 11:16:32 UTCcall?
sparrow
Fri, 31 Mar 2023 11:16:52 UTCyes that would be helpful
Ashish
Fri, 31 Mar 2023 11:17:23 UTCjoin huddle
sparrow
Fri, 31 Mar 2023 11:23:32 UTC{“code”:20008,“message”:“Search SQL execute error”,“error_detail”:“Error during planning: Projection references non-aggregate values: Expression tbl._timestamp could not be resolved from available columns: tbl.key, COUNT(UInt8(1))“}
Ashish
Fri, 31 Mar 2023 11:38:16 UTCCan you please share backend logs as well
Ashish
Fri, 31 Mar 2023 11:38:53 UTCI mean ZincObserve server logs
sparrow
Fri, 31 Mar 2023 11:41:34 UTC[2023-03-31T11:38:56Z INFO zincobserve::service::search::datafusion::exec] Query agg:_count took 2.464 seconds.
[2023-03-31T11:38:56Z INFO zincobserve::service::search::datafusion::exec] Query agg sql: select date_bin(interval ‘5 second’, to_timestamp_micros(“_timestamp”), to_timestamp(‘2001-01-01T00:00:00’)) AS key, count(*) AS num FROM tbl WHERE ((_timestamp >= 1680261833538000 AND _timestamp < 1680262733538000) ) GROUP BY key ORDER BY key
[2023-03-31T11:38:56Z INFO actix_web::middleware::logger] 10.244.31.9 “POST /api/default/default/_json HTTP/1.1" 200 69 “9208” “-” “Fluent-Bit” 0.005976
[2023-03-31T11:38:56Z INFO actix_web::middleware::logger] 10.244.18.51 “POST /api/default/default/_json HTTP/1.1" 200 68 “1850” “-” “Fluent-Bit” 0.002087
[2023-03-31T11:38:56Z INFO actix_web::middleware::logger] 10.244.18.51 “POST /api/default/default/_json HTTP/1.1" 200 68 “8552” “-” “Fluent-Bit” 0.004857
[2023-03-31T11:38:56Z INFO actix_web::middleware::logger] 10.244.31.9 “POST /api/default/default/_json HTTP/1.1" 200 69 “14241” “-” “Fluent-Bit” 0.029486
[2023-03-31T11:38:56Z ERROR zincobserve::service::search::datafusion::exec] aggs sql execute failed, session: Session { id: “3b970678-1409-4d6f-b5eb-e7c1a395320c”, data_type: Cache }, sql: select date_bin(interval ‘5 second’, to_timestamp_micros(“_timestamp”), to_timestamp(‘2001-01-01T00:00:00’)) AS key, count(*) AS num FROM tbl WHERE ((_timestamp >= 1680261833538000 AND _timestamp < 1680262733538000) ) GROUP BY key ORDER BY key, err: Plan(“Projection references non-aggregate values: Expression tbl._timestamp could not be resolved from available columns: tbl.key, COUNT(UInt8(1))“)
[2023-03-31T11:38:56Z ERROR zincobserve::service::search::cache] datafusion execute error: Error during planning: Projection references non-aggregate values: Expression tbl._timestamp could not be resolved from available columns: tbl.key, COUNT(UInt8(1))
[2023-03-31T11:38:56Z ERROR zincobserve::handler::http::request::search] search error: ErrorCode(SearchSQLExecuteError(“Error during planning: Projection references non-aggregate values: Expression tbl._timestamp could not be resolved from available columns: tbl.key, COUNT(UInt8(1))“))
[2023-03-31T11:38:56Z INFO actix_web::middleware::logger] 10.244.31.9 “POST /api/default/_search?type=logs HTTP/1.1” 500 205 “310" “
Hengfei
Fri, 31 Mar 2023 11:51:16 UTCchecking
Hengfei
Fri, 31 Mar 2023 12:05:37 UTCsorry, i can’t find the problem for this logs, Can you compress the directory `data` and give us for debug?
Hengfei
Fri, 31 Mar 2023 12:38:35 UTCget the same error, thanks for you data.
Hengfei
Fri, 31 Mar 2023 12:38:38 UTCi will debug for that.
Hengfei
Fri, 31 Mar 2023 14:10:02 UTCpublic.ecr.aws/zinclabs/zincobserve-dev:v0.3.1-75ec795
Hengfei
Fri, 31 Mar 2023 14:10:18 UTCsparrow we fixed the issue, you can use this image for test.
sparrow
Mon, 03 Apr 2023 04:45:17 UTCThanks Hengfei will check today with this image and let you in any case
sparrow
Mon, 03 Apr 2023 10:22:38 UTCI thinks it’s working fine now. But have one question, I thinks some logs getting missed from specific pods
Hengfei
Mon, 03 Apr 2023 10:25:41 UTCLike what?
sparrow
Mon, 03 Apr 2023 10:26:22 UTClike in pod logs I can see some error or some details but not visible on zinc
Ashish
Mon, 03 Apr 2023 10:26:52 UTCdo you have any functions
Ashish
Mon, 03 Apr 2023 10:27:12 UTCon the streams which may result in discarding ingested records?
sparrow
Mon, 03 Apr 2023 10:29:37 UTCNope
Hengfei
Mon, 03 Apr 2023 10:31:11 UTCAshish maybe we can let sparrow wait our next release, we will add more error response for ingestion.
Hengfei
Mon, 03 Apr 2023 10:31:32 UTCthey will see some error if we drop some records.
sparrow
Mon, 03 Apr 2023 10:35:54 UTCAny ETA for the same?
OpenObserve
Indexed 419 threads
OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI
View in App
sparrow
Fri, 31 Mar 2023 10:58:09 UTCHello Team, I was trying to integrate ZincObserver to our cluster, and it’s deployed successfully. But have few questions. 1. i’m not able to find the logs accordingly 2. Wants to send the data to s3 and fetch the data from s3 only 3. Want to apply delete policy for specific time interval How to achieve all these things?