Understanding Thread ID and WAL Feature
TLDR Karan was unsure about the use of 'thread_id' in a process. Ashish explained its function during data ingestion and introduced the ZO_FEATURE_PER_THREAD_LOCK feature to improve ingest performance.
Ashish
Thu, 14 Sep 2023 08:34:29 UTCwe use thread id (which in this case for every thread is 0) to decide if during ingestion if every thread should write to seperate file in WAL based on thread id
Ashish
Thu, 14 Sep 2023 08:35:08 UTCthe feature by default is disabled and can be enabled with ZO_FEATURE_PER_THREAD_LOCK
Karan
Thu, 14 Sep 2023 08:35:21 UTCNoted, thanks!
Ashish
Thu, 14 Sep 2023 08:35:33 UTCZO_FEATURE_PER_THREAD_LOCK false No default we share a lock for each thread for WAL, enable this option to create one lock for per thread, it improves ingest performance, but results in more small data files, which will be merged by compactor to create larger merged files. This is particularly helpful when you are ingesting high speed data in a single stream.
Ashish
Thu, 14 Sep 2023 08:35:42 UTC
OpenObserve
Indexed 419 threads
OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI
View in App
Karan
Thu, 14 Sep 2023 08:11:08 UTCNot sure if it's an error or I am misinterpreting something, but shouldn't `thread_id` be `0-7` (given there are 8 cores) ?