#openobserve

Discussing Log Level Metadata Storage and Restoration in ETCD

TLDR JinMing inquired about log level metadata storage in ETCD. Prabhat explained that even at PB scale ETCD usage should be low. They also advised on regular backups for data restoration.

Powered by Struct AI
7
3mo
Solved
Join the chat
Aug 14, 2023 (3 months ago)
JinMing
Photo of md5-46d1542107f070662960750eb3a2604f
JinMing
02:29 AM
Excuse me, guys. Our log level may reach PB level, how much metadata will be stored in ETCD? How can we better ensure the reliability of metadata?
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
02:30 AM
How much data do you have right now in OpenObserve?
JinMing
Photo of md5-46d1542107f070662960750eb3a2604f
JinMing
02:32 AM
At present, I only use it in the test environment, with very little metadata information and logs, and I can't judge the situation after going online.
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
03:08 AM
Only metadata like user details, dashboard, field list etc are stored in etcd. So even with PB scale etcd usage should be very low. So you plan to use it for logs, metrics and traces or just for logs?
JinMing
Photo of md5-46d1542107f070662960750eb3a2604f
JinMing
06:11 AM
Thank you for the information. I only plan to use it for logs.
06:40
JinMing
06:40 AM
There is another problem. If the data of etcd is accidentally lost, how should we restore the log information?
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
10:06 AM
Back up etcd regularly just like any other database and restore it