#openobserve

Discussing Log Level Metadata Storage and Restoration in ETCD

TLDR JinMing inquired about log level metadata storage in ETCD. Prabhat explained that even at PB scale ETCD usage should be low. They also advised on regular backups for data restoration.

3mo

Join the chat

Aug 14, 2023 (3 months ago)

Photo of md5-46d1542107f070662960750eb3a2604f

JinMing

02:29 AM

Excuse me, guys. Our log level may reach PB level, how much metadata will be stored in ETCD? How can we better ensure the reliability of metadata?

Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5

Prabhat

02:30 AM

How much data do you have right now in OpenObserve?

JinMing

02:32 AM

At present, I only use it in the test environment, with very little metadata information and logs, and I can't judge the situation after going online.

Prabhat

03:08 AM

Only metadata like user details, dashboard, field list etc are stored in etcd. So even with PB scale etcd usage should be very low. So you plan to use it for logs, metrics and traces or just for logs?

JinMing

06:11 AM

Thank you for the information. I only plan to use it for logs.

06:40

JinMing

06:40 AM

There is another problem. If the data of etcd is accidentally lost, how should we restore the log information?

Prabhat

10:06 AM

Back up etcd regularly just like any other database and restore it

OpenObserve

OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI

Indexed 406 threads (74% resolved)

Join Our Community

Similar Threads

Postgres Mode and etcd in Cluster Sync/State

Edgars inquired about the necessity of etcd in postgres mode. Hengfei confirmed its use and discussed potential future changes.

Discussion about Log Data as Big Data Storage

d daniel inquired about using logs for big data storage. Prabhat explained it's possible, but with limitations. No plans exist for ODBC/JDBC addition. The log data uses a custom DB engine.

4mo

Connectivity Testing Issues and Solutions with OpenObserve Router

Dylan experienced connectivity issues while testing OpenObserve router. Suggested by Prabhat regarding etcd issues and possibility of Istio service mesh causing connectivity problems. Reinstallation and increasing memory were also suggested but issue remained unresolved.