Optimizing OpenObserve with Vertical Scaling
TLDR Karan inquired about improving OpenObserve's performance through vertical scaling in local mode. Ashish responded that increased cores/memory would enhance its performance, but limits on single node deployment are yet to be defined.
Ashish
Thu, 14 Sep 2023 06:15:54 UTCHI Karan
Ashish
Thu, 14 Sep 2023 06:17:04 UTCin non-ha mode or local mode single executable plays all node roles like ingester , querier , compactor etc , please check
Ashish
Thu, 14 Sep 2023 06:19:59 UTCso you cant take advantage of multiple queriers , however having more cores/memory would let openobserve perform better
Ashish
Thu, 14 Sep 2023 06:20:44 UTCwe will get back to you on limits of single node deployment
Karan
Thu, 14 Sep 2023 06:22:28 UTCCheers, thanks! That would be super helpful! I am using m6a.4xlarge to test on my end, but anything from the maintainers would help as well.
OpenObserve
Indexed 419 threads
OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI
View in App
Karan
Thu, 14 Sep 2023 06:12:15 UTCHey folks. I've been experimenting with OpenObserve and it looks to be really promising so far. I've ingested a few million events locally (dummy logs) and playing around. I've a small question though. According to the docs: > Sled and Local disk > Use this mode for light usage and testing or if HA is not a requirement for you. (You could still ingest and search over 2 TB on a single machine per day. Does OpenObserve benefit with _Vertical Scaling_? As in, I figured that in the HA Mode, parallel query workers can work together on different parts of data and leader will aggregate. On the _local/single node_ mode, can I take advantage of more cores/memory? I'd like to stick with the local mode, because I think that eliminates the network I/O bottleneck of fetching files from S3, it's much more operationally easier for me to host it as well. So given that, how far can I stretch the system on a single node, by throwing more hardware to it? Thanks again, fantastic product and hoping it gains traction soon!