#general

Querying Local Storage Files and Ingesting Logs with OpenObserve

TLDR Rinshad asks about querying local storage files and ingesting logs. Prabhat suggests using parquet file format and tools like vector and fluentbit. Mark mentions Sublime's plugin for editing parquet files.

5mo

Join the chat

Jun 20, 2023 (5 months ago)

Photo of md5-023802ee0420f8b4bde4cee3631a5078

Rinshad

03:31 PM

Hi, I would like to query the local storage files directly using any utilities. Just like qurerying an SQLite DB file. For that can I know in which DB format openobserve is writing the stream data and its metadata in to local Disk, Is there any option currently we have in place to read the data directly from Disk itself from any external utility?

Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5

Prabhat

03:35 PM

OpenObserve stores data in parquet files. You can find a lot of tools to read parquet files directly. big data tools like trino, presto, spark, hadoop. small time tools like datafusion-cli, duckdb, and many more

Photo of md5-18020a76c76420f545aa508c19a6ec8d

Mark

09:01 PM

Even sublime has a plugin; just "edit" the file. Love the parquet format for this reason.

Jun 21, 2023 (5 months ago)

Rinshad

05:27 AM

Thanks for the support. So I assume that stream data files are in parquet file format inside the data/openobserve/stream/files directory and the metadata files in data/openobserve/db directory. is there any way to read the metadata files too.

Prabhat

01:51 PM

I assume that you are using single node mode. In single node mode OpenObserve uses sled for metadata. While we have not tried but you could try using https://git.deuxfleurs.fr/lx/sledcli or https://github.com/vi/sledtool

Rinshad

02:51 PM

Thank you.

02:53

Rinshad

02:53 PM

I was trying to ingest docker container logs and PostgreSQL logs . Can you help me with any reference to so this. I tried ingestion of xxxx.json.log from container logs directory just like the quickstart guide's json ingestion using curl but failed.

Prabhat

04:03 PM

Try using vector and configure docker logs https://vector.dev/docs/reference/configuration/sources/docker_logs/

Jun 22, 2023 (5 months ago)

Rinshad

10:46 AM

Thank you. I will check this. how about push the postgreSQL logs from filesystem?

Prabhat

12:30 PM

You should be able to point vector or fluentbit to the log file location of postgres and they will capture it. and send it to OpenObserve . Check https://vector.dev/docs/reference/configuration/sources/file/ or https://docs.fluentbit.io/manual/pipeline/inputs/tail

12:38

Prabhat

12:38 PM

Postgres log files are generally located at

/var/lib/postgresql/<version>/main/pg_log/postgresql-*.log

OpenObserve

OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI

Indexed 406 threads (74% resolved)

Join Our Community

Similar Threads

Troubleshooting Openobserve Setup with Azure Blob on K8S

David had difficulties setting up Openobserve with Azure Blob in Kubernetes. Prabhat provided guidance and setup values, informing David about possible data transmission delays, which resolved the issue.

Troubleshooting Openobserve Issue On Linux Machine

Sushma encountered difficulties viewing pods and creating multiple streams in Openobserve. Prabhat and Hengfei suggested upgrading the software, adjusting SQL syntax, and amending the configuration setup. Despite progress, the issue remains partially unresolved.

Troubleshooting Kubernetes Deployment and PostgreSQL Connection

Johnson faced issues fetching summary from a main page of a Kubernetes deployment, also observed PostgreSQL connection error on Azure, while the same setup was successful in AWS. Prabhat and Hengfei explored solutions but the problems persisted.

today

Resolving json logs interpretation issue in OpenObserve with Vector

Gabriel was facing a problem with OpenObserve interpreting json logs as plain text, using Vector for log routing. Ashish helped by suggesting adding a json parser to the Vector configuration, which solved the issue.

2mo

Troubleshooting Kafka Log Setup with OpenObserve

Ivan encountered an issue while setting up Kafka logs with OpenObserve and shared their vector config. Prabhat suggested a possible resource issue, but a final resolution wasn't determined.

4mo