#general

Querying Local Storage Files and Ingesting Logs with OpenObserve

TLDR Rinshad asks about querying local storage files and ingesting logs. Prabhat suggests using parquet file format and tools like vector and fluentbit. Mark mentions Sublime's plugin for editing parquet files.

Powered by Struct AI
11
5mo
Solved
Join the chat
Jun 20, 2023 (5 months ago)
Rinshad
Photo of md5-023802ee0420f8b4bde4cee3631a5078
Rinshad
03:31 PM
Hi, I would like to query the local storage files directly using any utilities. Just like qurerying an SQLite DB file. For that can I know in which DB format openobserve is writing the stream data and its metadata in to local Disk, Is there any option currently we have in place to read the data directly from Disk itself from any external utility?
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
03:35 PM
OpenObserve stores data in parquet files. You can find a lot of tools to read parquet files directly. big data tools like trino, presto, spark, hadoop. small time tools like datafusion-cli, duckdb, and many more
Mark
Photo of md5-18020a76c76420f545aa508c19a6ec8d
Mark
09:01 PM
Even sublime has a plugin; just "edit" the file. Love the parquet format for this reason.
Jun 21, 2023 (5 months ago)
Rinshad
Photo of md5-023802ee0420f8b4bde4cee3631a5078
Rinshad
05:27 AM
Thanks for the support. So I assume that stream data files are in parquet file format inside the data/openobserve/stream/files directory and the metadata files in data/openobserve/db directory. is there any way to read the metadata files too.
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
01:51 PM
I assume that you are using single node mode. In single node mode OpenObserve uses sled for metadata. While we have not tried but you could try using https://git.deuxfleurs.fr/lx/sledcli or https://github.com/vi/sledtool
Rinshad
Photo of md5-023802ee0420f8b4bde4cee3631a5078
Rinshad
02:51 PM
Thank you.
02:53
Rinshad
02:53 PM
I was trying to ingest docker container logs and PostgreSQL logs . Can you help me with any reference to so this. I tried ingestion of xxxx.json.log from container logs directory just like the quickstart guide's json ingestion using curl but failed.
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
04:03 PM
Jun 22, 2023 (5 months ago)
Rinshad
Photo of md5-023802ee0420f8b4bde4cee3631a5078
Rinshad
10:46 AM
Thank you. I will check this. how about push the postgreSQL logs from filesystem?
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
12:30 PM
You should be able to point vector or fluentbit to the log file location of postgres and they will capture it. and send it to OpenObserve . Check https://vector.dev/docs/reference/configuration/sources/file/ or https://docs.fluentbit.io/manual/pipeline/inputs/tail
12:38
Prabhat
12:38 PM
Postgres log files are generally located at
/var/lib/postgresql/<version>/main/pg_log/postgresql-*.log

OpenObserve

OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI

Indexed 406 threads (74% resolved)

Join Our Community