Issues with Query Operation after Software Upgrade
TLDR Chris had issues with specific queries after upgrading software version from 0.5.2 to 0.6.1. Hengfei tried to reproduce the error but was unsuccessful. The issue remains unresolved.
Sep 19, 2023 (2 months ago)
Chris
02:13 PMSep 19 15:01:14 oo openobserve[2176]: [2023-09-19T14:01:14Z ERROR openobserve::service::search::grpc::storage] datafusion execute error: Arrow error: External error: Arrow: Parquet argument error: Parquet error: StructArrayReader out of sync in read_records, expected 64 read, got 0
Sep 19 15:01:14 oo openobserve[2176]: [2023-09-19T14:01:14Z ERROR openobserve::service::search] search->grpc: node: 1, search err: Status { code: Internal, message: "{\"code\":20008,\"inner\":\"Arrow error: External error: Arrow: Parquet argument error: Parquet error: StructArrayReader out of sync in read_records, expected 64 read, got 0\",\"message\":\"Search SQL execute error\"}", metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 19 Sep 2023 14:01:14 GMT", "content-length": "0"} }, source: None }
Sep 19 15:01:14 oo openobserve[2176]: [2023-09-19T14:01:14Z ERROR openobserve::handler::http::request::search] search error: ErrorCode(SearchSQLExecuteError("Arrow error: External error: Arrow: Parquet argument error: Parquet error: StructArrayReader out of sync in read_records, expected 64 read, got 0"))
Hengfei
02:20 PMHengfei
02:21 PMChris
02:21 PMChris
02:23 PMHengfei
02:24 PMChris
02:27 PMHengfei
02:28 PMHengfei
02:28 PMHengfei
02:29 PMHengfei
02:30 PMHengfei
02:30 PMChris
02:31 PMHengfei
02:31 PMChris
02:42 PMHengfei
02:45 PMHengfei
02:45 PMSep 20, 2023 (2 months ago)
Chris
03:49 PMHengfei
11:41 PMSep 21, 2023 (2 months ago)
Chris
04:12 PMTimeline:
7 Aug - standalone non-containerised instance created, 0.5.1, sled/s3
7 Aug - stream jo created, fts message,content
8 Aug - stream jb created, fts message
22 Aug - stream ex created, fts message
23-Aug - stream ha created, fts message,http_request
24-Aug - stream ap created, fts message
25-Aug - upgrade to 0.5.2
7-Sep - stream er created, fts message
19-Sep - upgrade to 0.6.1
All of the streams were created on those dates, but backfilled a few weeks/months, so the early dates of the log entries do not necessarily reflect when they were ingested.
All streams can be queried over 6 months without any query terms.
The jo & ha streams fail when queried back before 25 Aug with a search term.
The thing that stands out is that both of those two streams have a second fts field...
OpenObserve
Indexed 406 threads (74% resolved)
Similar Threads
OpenObserve Upgrade to v0.6.2 Causes Errors
Joaquin faced issues with OpenObserve after upgrading it. Prabhat provided troubleshooting advice but the issue was ultimately identified and fixed in v0.6.3 by Hengfei. After upgrading, Joaquin confirmed the issue was resolved.
Troubleshooting openobserve HA Self-hosted Set Up Error
Dhananjay faced an issue querying logs with their openobserve HA setup. Ashish, Prabhat, and Joaquin provided troubleshooting steps. The issue was resolved by increasing the read-timeout.
Trouble Running OpenObserve with Docker and GCS Storage
Jay had trouble running openobserve with Docker and GCS. Prabhat suggested adding different environments. After resolving this, Jay still had data storage issues, which were clarified by Hengfei.
Fixing openobserve with rust tracing and opentelemetry-otlp issue
Gunnar had problems with openobserve using rust tracing and opentelemetry-otlp. Ashish and West provided solutions to resolve the issue and discussed matter related to gRPC and HTTP.
Troubleshooting Metrics Display Issue in On-Prem 0.5.2
Chris is having issues displaying metrics even after successful data ingestion and receiving 200 responses. Hengfei suggested checking the query and time range, then advised creating an issue for it.