Discussing Features and Compatibility of Log Processing Component
TLDR Tony queried about Log processing component, comparing it with Loki/Elastic/Splunk. Ashish clarified its usage and planned updates. Tony also asked about Grafana compatibility, which Prabhat confirmed. last inquired about storage setup, and Prabhat described available modes.
Ashish
Mon, 03 Jul 2023 02:43:50 UTCWe don't use inverted index...rather while searching we use brute force search as you mentioned
Ashish
Mon, 03 Jul 2023 02:44:30 UTCWe use data partitioning to reduce scan volumn size based on time range for query
Tony
Mon, 03 Jul 2023 02:44:33 UTCYeah same with the people who all use Clickhouse :slightly_smiling_face:
Tony
Mon, 03 Jul 2023 02:44:45 UTCreasonable approach
Ashish
Mon, 03 Jul 2023 02:45:01 UTCWe also support custom partitioning keys
Ashish
Mon, 03 Jul 2023 02:45:30 UTCWe will be adding adaptive partitioning based on query usage in future
Ashish
Mon, 03 Jul 2023 02:46:46 UTCDo let us know any feedback or suggestions
Ashish
Mon, 03 Jul 2023 02:47:03 UTCAlso if you face any issue ..we would be glad to assist
Tony
Mon, 03 Jul 2023 02:51:07 UTCDoes it work with Grafana?
Tony
Mon, 03 Jul 2023 02:51:12 UTCI am currently using Prometheus + Loki
Prabhat
Mon, 03 Jul 2023 04:22:43 UTCThere is a plug-in for grafana that u can use.
last
Mon, 03 Jul 2023 04:58:17 UTCwhat kind of storage is used ? (i see in readme its using object storage) i was doing a setup from source dint require any database setup could you provide more detail on storage
Prabhat
Mon, 03 Jul 2023 04:58:59 UTCYou can either use local disk storage in single node mode or s3 in HA mode
OpenObserve
Indexed 419 threads
OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI
View in App
Tony
Mon, 03 Jul 2023 02:05:31 UTCI see: "You should give a time range for each query or it will scan all data, it is a very expensive operate. So it seems like the Log processing component is more like Loki than Elastic/Splunk that supports inverted indices?