Troubleshooting Metrics Display Issue in On-Prem 0.5.2
TLDR Chris is having issues displaying metrics even after successful data ingestion and receiving 200 responses. Hengfei suggested checking the query and time range, then advised creating an issue for it.
Hengfei
Thu, 14 Sep 2023 11:13:57 UTCselect a bigger time range, like 1 hour, 2 hour? and try again?
Chris
Thu, 14 Sep 2023 11:14:48 UTCdone that.... always no data... the timestamps are only a minute or two behind when they are submitted, and i am looking soon after that
Hengfei
Thu, 14 Sep 2023 11:16:26 UTCCan you give some samples of your ingested data?
Chris
Thu, 14 Sep 2023 11:34:17 UTCsometimes it works, sometimes it doesn't... struggling to find the pattern :disappointed:
Chris
Thu, 14 Sep 2023 11:35:32 UTCwhat does the `[PromExpr::Binary] either lhs or rhs vector is found to be empty` message mean and what can provoke it?
Hengfei
Thu, 14 Sep 2023 11:35:56 UTCWhat is your query?
Chris
Thu, 14 Sep 2023 11:37:21 UTCsimply `gaugename{}`
Chris
Thu, 14 Sep 2023 11:45:42 UTCi think it is failing when the stream name contains a hyphen e.g. z-bytes1
Chris
Thu, 14 Sep 2023 11:52:14 UTCyup.... stream names of bytes1 works, z_bytes1 works, but z-bytes1 always throws those errors in the log and returns no data
Hengfei
Fri, 15 Sep 2023 03:35:39 UTCokay, maybe, like me check. can you create an issue for this let us can track it?
Chris
Fri, 15 Sep 2023 08:32:19 UTCsure, have added it to my list of issues to create :grin:
OpenObserve
Indexed 419 threads
OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI
View in App
Chris
Thu, 14 Sep 2023 11:11:54 UTC[on-prem 0.5.2] i am struggling getting metrics to display.... values are being submitted using /api/xxx/ingest/metrics/_json, and are getting 200 responses... the streams are created, and the doc counts are increasing... timestamps are being supplied in milliseconds.... but the gui is always reporting No Data no matter what time range i select..... and the querier nodes report lots of `[PromExpr::Binary] either lhs or rhs vector is found to be empty` , then lines like ```[2023-09-14T11:03:06Z INFO openobserve::service::promql::search] promql->search->grpc: result node: 2, need_wal: false, took: 7, files: 0, scan_size: 0 [2023-09-14T11:03:06Z INFO openobserve::service::promql::search] promql->search->grpc: result node: 3, need_wal: true, took: 9, files: 0, scan_size: 0 [2023-09-14T11:03:06Z INFO openobserve::service::promql::search] promql->search->result: took: 11, file_count: 0, scan_size: 0``` followed by a 200 response.... and yet some test metric streams with very similar data work.... i'm running out of ideas where to look.... any ideas?