#openobserve

Issues with Query Operation after Software Upgrade

TLDR Chris had issues with specific queries after upgrading software version from 0.5.2 to 0.6.1. Hengfei tried to reproduce the error but was unsuccessful. The issue remains unresolved.

Powered by Struct AI
Sep 19, 2023 (2 months ago)
Chris
Photo of md5-7d2796ababc14477f4c09abc2e8edb05
Chris
02:13 PM
just upgraded 0.5.2 -> 0.6.1 on-prem, sled/s3..... created the stream directory, upgrade seemed to work, restarted oo a couple of times... ingest running ok.... BUT.... with some log streams a query without any search criteria works, but specifying field=value causes a sql error and the following log messages
Sep 19 15:01:14 oo openobserve[2176]: [2023-09-19T14:01:14Z ERROR openobserve::service::search::grpc::storage] datafusion execute error: Arrow error: External error: Arrow: Parquet argument error: Parquet error: StructArrayReader out of sync in read_records, expected 64 read, got 0
Sep 19 15:01:14 oo openobserve[2176]: [2023-09-19T14:01:14Z ERROR openobserve::service::search] search->grpc: node: 1, search err: Status { code: Internal, message: "{\"code\":20008,\"inner\":\"Arrow error: External error: Arrow: Parquet argument error: Parquet error: StructArrayReader out of sync in read_records, expected 64 read, got 0\",\"message\":\"Search SQL execute error\"}", metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 19 Sep 2023 14:01:14 GMT", "content-length": "0"} }, source: None }
Sep 19 15:01:14 oo openobserve[2176]: [2023-09-19T14:01:14Z ERROR openobserve::handler::http::request::search] search error: ErrorCode(SearchSQLExecuteError("Arrow error: External error: Arrow: Parquet argument error: Parquet error: StructArrayReader out of sync in read_records, expected 64 read, got 0"))
Hengfei
Photo of md5-c30bb074b7d997d2cd6e689678b65dc1
Hengfei
02:20 PM
yes, other users report this too, but only for search old data, right?
02:21
Hengfei
02:21 PM
And 0.5.2 how no this error.
Chris
Photo of md5-7d2796ababc14477f4c09abc2e8edb05
Chris
02:21 PM
0.5.2 was ok.... think it is only old data but let me check
02:23
Chris
02:23 PM
think it may be data older than about 3-4 weeks
Hengfei
Photo of md5-c30bb074b7d997d2cd6e689678b65dc1
Hengfei
02:24 PM
yes, the user report for about 4 weeks ago, recently three weeks has no problem
Chris
Photo of md5-7d2796ababc14477f4c09abc2e8edb05
Chris
02:27 PM
please tell me the data is safe... :hot_face:
Hengfei
Photo of md5-c30bb074b7d997d2cd6e689678b65dc1
Hengfei
02:28 PM
the data is safe, but only 0.5.2 can search it. new version can't search the data. and as the other user test only for a special time range.
02:28
Hengfei
02:28 PM
he can position only the day.
02:29
Hengfei
02:29 PM
other days search no problem.
02:30
Hengfei
02:30 PM
but i can't reproduce it, the user deny provide the data to me for test.
02:30
Hengfei
02:30 PM
Can you try to find which day has the problem, and provide some data to me for test?
Chris
Photo of md5-7d2796ababc14477f4c09abc2e8edb05
Chris
02:31 PM
let me see what i can find out, but i can't send raw data
Hengfei
Photo of md5-c30bb074b7d997d2cd6e689678b65dc1
Hengfei
02:31 PM
understand
Chris
Photo of md5-7d2796ababc14477f4c09abc2e8edb05
Chris
02:42 PM
looking at one stream, only 24 Aug fails..... on another stream 24 Aug and earlier fails
Hengfei
Photo of md5-c30bb074b7d997d2cd6e689678b65dc1
Hengfei
02:45 PM
yes, 24 Aug, the day he upgraded to 0.5.2, the user said that.
02:45
Hengfei
02:45 PM
Okay, i have some idea, i will test, thank you.
Sep 20, 2023 (2 months ago)
Chris
Photo of md5-7d2796ababc14477f4c09abc2e8edb05
Chris
03:49 PM
Any news on this?
Hengfei
Photo of md5-c30bb074b7d997d2cd6e689678b65dc1
Hengfei
11:41 PM
no, I tested 0.5.1 to 0.5.2 to 0.6.1, can’t reproduce the error
Sep 21, 2023 (2 months ago)
Chris
Photo of md5-7d2796ababc14477f4c09abc2e8edb05
Chris
04:12 PM
Additional info that might help...
Timeline:
7 Aug - standalone non-containerised instance created, 0.5.1, sled/s3
7 Aug - stream jo created, fts message,content
8 Aug - stream jb created, fts message
22 Aug - stream ex created, fts message
23-Aug - stream ha created, fts message,http_request
24-Aug - stream ap created, fts message
25-Aug - upgrade to 0.5.2
7-Sep - stream er created, fts message
19-Sep - upgrade to 0.6.1

All of the streams were created on those dates, but backfilled a few weeks/months, so the early dates of the log entries do not necessarily reflect when they were ingested.

All streams can be queried over 6 months without any query terms.
The jo & ha streams fail when queried back before 25 Aug with a search term.
The thing that stands out is that both of those two streams have a second fts field...

OpenObserve

OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI

Indexed 406 threads (74% resolved)

Join Our Community