Troubleshooting Metrics Display Issue in On-Prem 0.5.2

TLDR Chris is having issues displaying metrics even after successful data ingestion and receiving 200 responses. Hengfei suggested checking the query and time range, then advised creating an issue for it.

Photo of Chris
Chris
Thu, 14 Sep 2023 11:11:54 UTC

[on-prem 0.5.2] i am struggling getting metrics to display.... values are being submitted using /api/xxx/ingest/metrics/_json, and are getting 200 responses... the streams are created, and the doc counts are increasing... timestamps are being supplied in milliseconds.... but the gui is always reporting No Data no matter what time range i select..... and the querier nodes report lots of `[PromExpr::Binary] either lhs or rhs vector is found to be empty` , then lines like ```[2023-09-14T11:03:06Z INFO openobserve::service::promql::search] promql->search->grpc: result node: 2, need_wal: false, took: 7, files: 0, scan_size: 0 [2023-09-14T11:03:06Z INFO openobserve::service::promql::search] promql->search->grpc: result node: 3, need_wal: true, took: 9, files: 0, scan_size: 0 [2023-09-14T11:03:06Z INFO openobserve::service::promql::search] promql->search->result: took: 11, file_count: 0, scan_size: 0``` followed by a 200 response.... and yet some test metric streams with very similar data work.... i'm running out of ideas where to look.... any ideas?

Photo of Hengfei
Hengfei
Thu, 14 Sep 2023 11:13:57 UTC

select a bigger time range, like 1 hour, 2 hour? and try again?

Photo of Chris
Chris
Thu, 14 Sep 2023 11:14:48 UTC

done that.... always no data... the timestamps are only a minute or two behind when they are submitted, and i am looking soon after that

Photo of Hengfei
Hengfei
Thu, 14 Sep 2023 11:16:26 UTC

Can you give some samples of your ingested data?

Photo of Chris
Chris
Thu, 14 Sep 2023 11:34:17 UTC

sometimes it works, sometimes it doesn't... struggling to find the pattern :disappointed:

Photo of Chris
Chris
Thu, 14 Sep 2023 11:35:32 UTC

what does the `[PromExpr::Binary] either lhs or rhs vector is found to be empty` message mean and what can provoke it?

Photo of Hengfei
Hengfei
Thu, 14 Sep 2023 11:35:56 UTC

What is your query?

Photo of Chris
Chris
Thu, 14 Sep 2023 11:37:21 UTC

simply `gaugename{}`

Photo of Chris
Chris
Thu, 14 Sep 2023 11:45:42 UTC

i think it is failing when the stream name contains a hyphen e.g. z-bytes1

Photo of Chris
Chris
Thu, 14 Sep 2023 11:52:14 UTC

yup.... stream names of bytes1 works, z_bytes1 works, but z-bytes1 always throws those errors in the log and returns no data

Photo of Hengfei
Hengfei
Fri, 15 Sep 2023 03:35:39 UTC

okay, maybe, like me check. can you create an issue for this let us can track it?

Photo of Chris
Chris
Fri, 15 Sep 2023 08:32:19 UTC

sure, have added it to my list of issues to create :grin: