Issues with Query Operation after Software Upgrade

just upgraded 0.5.2 -> 0.6.1 on-prem, sled/s3..... created the stream directory, upgrade seemed to work, restarted oo a couple of times... ingest running ok.... BUT.... with some log streams a query without any search criteria works, but specifying field=value causes a sql error and the following log messages

Sep 19 15:01:14 oo openobserve[2176]: [2023-09-19T14:01:14Z ERROR openobserve::service::search::grpc::storage] datafusion execute error: Arrow error: External error: Arrow: Parquet argument error: Parquet error: StructArrayReader out of sync in read_records, expected 64 read, got 0
Sep 19 15:01:14 oo openobserve[2176]: [2023-09-19T14:01:14Z ERROR openobserve::service::search] search->grpc: node: 1, search err: Status { code: Internal, message: "{\"code\":20008,\"inner\":\"Arrow error: External error: Arrow: Parquet argument error: Parquet error: StructArrayReader out of sync in read_records, expected 64 read, got 0\",\"message\":\"Search SQL execute error\"}", metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 19 Sep 2023 14:01:14 GMT", "content-length": "0"} }, source: None }
Sep 19 15:01:14 oo openobserve[2176]: [2023-09-19T14:01:14Z ERROR openobserve::handler::http::request::search] search error: ErrorCode(SearchSQLExecuteError("Arrow error: External error: Arrow: Parquet argument error: Parquet error: StructArrayReader out of sync in read_records, expected 64 read, got 0"))

yes, other users report this too, but only for search old data, right?

0.5.2 was ok.... think it is only old data but let me check

yes， the user report for about 4 weeks ago, recently three weeks has no problem

please tell me the data is safe... :hot_face:

the data is safe, but only 0.5.2 can search it. new version can't search the data. and as the other user test only for a special time range.

let me see what i can find out, but i can't send raw data

understand

looking at one stream, only 24 Aug fails..... on another stream 24 Aug and earlier fails

yes, 24 Aug, the day he upgraded to 0.5.2, the user said that.

Any news on this?

no, I tested 0.5.1 to 0.5.2 to 0.6.1, can’t reproduce the error

Additional info that might help...
Timeline:
7 Aug - standalone non-containerised instance created, 0.5.1, sled/s3
7 Aug - stream jo created, fts message,content
8 Aug - stream jb created, fts message
22 Aug - stream ex created, fts message
23-Aug - stream ha created, fts message,http_request
24-Aug - stream ap created, fts message
25-Aug - upgrade to 0.5.2
7-Sep - stream er created, fts message
19-Sep - upgrade to 0.6.1

All of the streams were created on those dates, but backfilled a few weeks/months, so the early dates of the log entries do not necessarily reflect when they were ingested.

All streams can be queried over 6 months without any query terms.
The jo & ha streams fail when queried back before 25 Aug with a search term.
The thing that stands out is that both of those two streams have a second fts field...