Utilizing VRL for Log Grouping and Searching in ETL Jobs
TLDR West requested help with log group and searching within an ETL context. Prabhat recommended using VRL at the ingestion level and clarified its limitations.
Prabhat
Wed, 27 Sep 2023 17:23:04 UTCYou could use VRL during search if that helps. It is considerably slower but it should provide you with what you need.
West
Wed, 27 Sep 2023 17:26:21 UTCWill it be faster if we apply VRL at ingestion level ? pls , do you have any example that will help us more
Prabhat
Wed, 27 Sep 2023 17:27:00 UTCYes, using VRL at ingestion time is the recommended way to use it for now while we work on making VRL performant at query time.
Prabhat
Wed, 27 Sep 2023 17:27:59 UTCCheck this for an example -
West
Wed, 27 Sep 2023 17:34:53 UTCI understand VRL function will takes log record/row as an input and it will help to enrich the log entry as an iterative process , pls correct me if I am wrong. My requirement is something like this, select all log entries based on some attribute -> on top of this result perform one more select (selection on top of the result)
Prabhat
Wed, 27 Sep 2023 17:35:29 UTCgive me an example
Prabhat
Wed, 27 Sep 2023 17:35:41 UTCenrichment is just one use case for VRL
West
Wed, 27 Sep 2023 17:52:03 UTCWe have some ETL jobs which will emit logs with unique job id with start time (when job started), similarly we will have log entry with end time also (when job ends), these jobs are multi step jobs for each step we would have status also , If I need to get the job completion time based on start time and end time of job, How do I get . Example Logs something like below startTime=2023-09-21T11:48:17.672252900, jobName=Producer, userContext=xyz, Steps=1,jobId=bf0db59b-6bcd-4b9a-b0a2-bc8ddccd1fb3,status=progressing jobName=Producer, userContext=xyz, Steps=2, jobId=bf0db59b-6bcd-4b9a-b0a2-bc8ddccd1fb3,status=progressing jobName=Producer, userContext=xyz, Steps=3, jobId=bf0db59b-6bcd-4b9a-b0a2-bc8ddccd1fb3,status=progressing jobName=Producer, userContext=xyz, Steps=4, jobId=bf0db59b-6bcd-4b9a-b0a2-bc8ddccd1fb3,status=progressing endTime=2023-09-21T12:48:17.672252900, jobName=Producer, userContext=xyz, Steps=3, jobId=bf0db59b-6bcd-4b9a-b0a2-bc8ddccd1fb3,status=complete
Prabhat
Wed, 27 Sep 2023 19:57:31 UTCis this a single log record? or multiple records?
West
Thu, 28 Sep 2023 04:00:42 UTCMultiple
Prabhat
Thu, 28 Sep 2023 04:24:52 UTCyou will need subqueries and joins that OpenObserve does not support yet. We do have plans to support these, but will take a bit.
West
Thu, 28 Sep 2023 04:25:38 UTCThank you
West
Thu, 28 Sep 2023 04:26:44 UTCCan we deal with VRL for this use case ?
Prabhat
Thu, 28 Sep 2023 04:28:02 UTCno. VRL works at a single record level. It can take 1 record in and 1 record out. or 1 record in and multiple records out. It can't do multiple records in and 1 record out yet.
OpenObserve
Indexed 419 threads
OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI
View in App
West
Wed, 27 Sep 2023 17:21:59 UTCHi Team , is there any way to group and search the logs based on some attribute and keep them as a temporary view or buffer and perform search operation on the view?