OpenObserve Errors and How to Fix Them

TLDR Ilya encountered errors while using OpenObserve. Ashish suggested using the dev image and provided instructions to clean up data. After implementing the solution, the errors were resolved.

Photo of Ilya
Ilya
Fri, 09 Jun 2023 12:32:14 UTC

Hey guys! I just set up OpenObserve as our production logging system, and it’s pretty neat! I’m glad to see another cool Rust development. Now, I’m encountering two errors while trying to execute a SELECT * from logs query: ```"External error: Arrow error: Json error: failed to parse 18432951799850641000 as Int64"``` ```"Arrow error: Json error: expected string got false"``` It seems like something under the hood is attempting to convert our log entries to a specified schema, but the logs are not strictly typed. Is there any workaround for this issue?

Photo of Ashish
Ashish
Fri, 09 Jun 2023 12:39:15 UTC

Hi Ilya..this is a bug in current release

Photo of Ashish
Ashish
Fri, 09 Jun 2023 12:39:51 UTC

you can use dev image to proceed or we would make a release next week with fixes

Photo of Ashish
Ashish
Fri, 09 Jun 2023 12:40:54 UTC

public.ecr.aws/zinclabs/openobserve-dev:v0.4.7-155f766

Photo of Ashish
Ashish
Fri, 09 Jun 2023 12:41:06 UTC

Can you try this image

Photo of Ilya
Ilya
Fri, 09 Jun 2023 12:41:17 UTC

Ashish gonna try rn

Photo of Ashish
Ashish
Fri, 09 Jun 2023 12:41:25 UTC

Also..if possible clean up existing data

Photo of Ashish
Ashish
Fri, 09 Jun 2023 12:41:34 UTC

ingest data & test

Photo of Ilya
Ilya
Fri, 09 Jun 2023 12:44:14 UTC

Ashish could you please led me on how can I clean minio storage while using Helm-deployed openobserve?

Photo of Ashish
Ashish
Fri, 09 Jun 2023 12:47:31 UTC

do you have MinIO Client (mc)

Photo of Ashish
Ashish
Fri, 09 Jun 2023 12:48:06 UTC

you can use that to mc rm --force --recursive myminio/mybucket/*

Photo of Hengfei
Hengfei
Fri, 09 Jun 2023 12:48:40 UTC

Actually, you can use the UI delete the stream.

Photo of Ilya
Ilya
Fri, 09 Jun 2023 12:48:46 UTC

oh.. my bucket is empty so looks like something is misconfigured

Photo of Hengfei
Hengfei
Fri, 09 Jun 2023 12:48:52 UTC

it will auto delete data in storage.

Photo of Ashish
Ashish
Fri, 09 Jun 2023 12:49:20 UTC

from Left Menu --> Streams

Photo of Ashish
Ashish
Fri, 09 Jun 2023 12:49:28 UTC

it will list call your streams

Photo of Ashish
Ashish
Fri, 09 Jun 2023 12:49:42 UTC

delete the ones you want to cleanup

Photo of Ilya
Ilya
Fri, 09 Jun 2023 12:51:35 UTC

Updated to aforementioned version, ingested some logs, the error now is: ```"Arrow error: Json error: failed to parse \"unknown\" as Int64"```

Photo of Ashish
Ashish
Fri, 09 Jun 2023 12:55:02 UTC

this is during ingestion?

Photo of Ashish
Ashish
Fri, 09 Jun 2023 12:55:25 UTC

this is single node deployment right?

Photo of Ashish
Ashish
Fri, 09 Jun 2023 12:55:52 UTC

in openobserv dir you should have /data/ dir

Photo of Ashish
Ashish
Fri, 09 Jun 2023 12:56:01 UTC

can you please clean up that as well

Photo of Ilya
Ilya
Fri, 09 Jun 2023 12:57:10 UTC

that’s helm k8s deployment with minio

Photo of Ilya
Ilya
Fri, 09 Jun 2023 12:57:32 UTC

that error appears while searching logs (`/api/default/_search?type=logs`)

Photo of Ashish
Ashish
Fri, 09 Jun 2023 12:59:45 UTC

delete the cluster & delete PV named like data-zo1-openobserve-ingester-0

Photo of Ashish
Ashish
Fri, 09 Jun 2023 13:00:42 UTC

since your wal has corrupted data

Photo of Ashish
Ashish
Fri, 09 Jun 2023 13:00:52 UTC

you are facing this issue