TLDR West requested help with log group and searching within an ETL context. Prabhat recommended using VRL at the ingestion level and clarified its limitations.
You could use VRL during search if that helps. It is considerably slower but it should provide you with what you need.
Will it be faster if we apply VRL at ingestion level ? pls , do you have any example that will help us more
Yes, using VRL at ingestion time is the recommended way to use it for now while we work on making VRL performant at query time.
Check this for an example -
I understand VRL function will takes log record/row as an input and it will help to enrich the log entry as an iterative process , pls correct me if I am wrong. My requirement is something like this, select all log entries based on some attribute -> on top of this result perform one more select (selection on top of the result)
give me an example
enrichment is just one use case for VRL
We have some ETL jobs which will emit logs with unique job id with start time (when job started), similarly we will have log entry with end time also (when job ends), these jobs are multi step jobs for each step we would have status also , If I need to get the job completion time based on start time and end time of job, How do I get . Example Logs something like below startTime=2023-09-21T11:48:17.672252900, jobName=Producer, userContext=xyz, Steps=1,jobId=bf0db59b-6bcd-4b9a-b0a2-bc8ddccd1fb3,status=progressing jobName=Producer, userContext=xyz, Steps=2, jobId=bf0db59b-6bcd-4b9a-b0a2-bc8ddccd1fb3,status=progressing jobName=Producer, userContext=xyz, Steps=3, jobId=bf0db59b-6bcd-4b9a-b0a2-bc8ddccd1fb3,status=progressing jobName=Producer, userContext=xyz, Steps=4, jobId=bf0db59b-6bcd-4b9a-b0a2-bc8ddccd1fb3,status=progressing endTime=2023-09-21T12:48:17.672252900, jobName=Producer, userContext=xyz, Steps=3, jobId=bf0db59b-6bcd-4b9a-b0a2-bc8ddccd1fb3,status=complete
is this a single log record? or multiple records?
Multiple
you will need subqueries and joins that OpenObserve does not support yet. We do have plans to support these, but will take a bit.
Thank you
Can we deal with VRL for this use case ?
no. VRL works at a single record level. It can take 1 record in and 1 record out. or 1 record in and multiple records out. It can't do multiple records in and 1 record out yet.
West
Wed, 27 Sep 2023 17:21:59 UTCHi Team , is there any way to group and search the logs based on some attribute and keep them as a temporary view or buffer and perform search operation on the view?