Elasticsearch vs Zinc for 40TB Dataset
TLDR Josh asks if zinc works for a 40TB dataset. Prabhat replies that zinc is not suitable due to single node limitation but suggests using zinc-observe for petabyte scale data.
Josh
Mon, 06 Feb 2023 00:10:23 UTCI have a 40TB / 10B document dataset
Josh
Mon, 06 Feb 2023 00:10:35 UTCdoes zinc work for this scale?
Prabhat
Mon, 06 Feb 2023 00:11:15 UTCIs this for log search or app search?
Josh
Mon, 06 Feb 2023 00:20:56 UTCapp search i guess
Prabhat
Mon, 06 Feb 2023 00:24:19 UTCzinc currently can run only on a single node. Supporting 40 TB on a single node does not really work. So zinc might work well for your app search use case. However if you had log data that you wanted to search you could use zinc-observe (
OpenObserve
Indexed 419 threads
OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI
View in App
Josh
Mon, 06 Feb 2023 00:10:04 UTCI have some basic qqs for elasticsearch vs zinc