Improving Queries Performance in Large Data
TLDR Joaquin sought advice to improve query speed on a large dataset. Prabhat suggested using partitioning to reduce the search space and increase speed.
Prabhat
Fri, 06 Oct 2023 10:56:29 UTCYou should leverage partitioning
Prabhat
Fri, 06 Oct 2023 10:58:25 UTCE.g. you can partition by namespace. After you partition by namespace you can use namespace as a search parameter before searching anything else. This will increase performance.
Prabhat
Fri, 06 Oct 2023 10:58:56 UTCThe idea is that you reduce the search space.
Prabhat
Fri, 06 Oct 2023 11:00:12 UTCThis will work for new data going forward after enabling partitioning.
Joaquin
Fri, 06 Oct 2023 12:21:48 UTCPerfect, thanks, I'll check which field gives me the best partitioning
OpenObserve
Indexed 419 threads
OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI
View in App
Joaquin
Fri, 06 Oct 2023 09:27:44 UTCHi folks! I would like any pointers that you might have regarding increasing the performance of queries. Let me describe the situation, right now we're ingesting about 15TB of logs per month and searching logs for anything past a few hours is super slow. I've tried this: • Increasing resources for querier nodes, right now I'm running 4 instances with 4vCPU and 10GB RAM on K8s. • Adding a persitent volume of 50GB for disk cache and increased the percentage from 50% utilization to 48GB. Do you have any suggestion on how to improve it?