Inner Workings of OpenObserve VRL Function

TLDR Mik inquired about the VRL function behavior in OpenObserve logs. Prabhat explained the system automatically flattens structures, a feature that won't change in the future.

Photo of Mik
Mik
Mon, 02 Oct 2023 04:15:56 UTC

Hi everyone! Is this the expected behavior of the VRL function in the open observe logs? VRL function ```.log = parse_json!(.message_message) del(.message_message) .``` VRL playground output: ```{ "_timestamp": 1695956833809000, "id": "37821101218481254744296987650977054435587665849865928704", "log": { "applicationName": "sbc-outbound", "event": { "message": "Refresh RTP Engine connections: 10.100.12.93" }, "eventTime": 1695956829, "instanceName": "sip-little-woodpecker", "level": "info" }, "loggroup": "/log/sbc-sip/sbc-log", "logstream": "sip-little-woodpecker", ... "requestid": "11111xxxx-fe77-481c-b60b-73a33a1d9602", "subscriptionfilters": "[\"CloudwatchOpenobserveIngestStack-SubscriptionFilterXXXXX-wZGu0KhDhTIC\"]", "timestamp": 1695956833809 }``` OpenObserve logs: ```{ _timestamp:1695956113809000, id:37821085161944711802248324874853977452967836770446606336, log_applicationname:sbc-outbound, log_event_message:Refresh RTP Engine connections: 10.100.12.93, log_eventtime:1695956109, log_instancename:sip-little-woodpecker, log_level:info, "loggroup": "/log/sbc-sip/sbc-log", "logstream": "sip-little-woodpecker", ... "requestid": "11111xxxx-fe77-481c-b60b-73a33a1d9602", "subscriptionfilters": "[\"CloudwatchOpenobserveIngestStack-SubscriptionFilterXXXXX-wZGu0KhDhTIC\"]", "timestamp": 1695956833809 }``` OpenObserve will automatically append an `underscore` for nested JSON object keys?

Photo of Prabhat
Prabhat
Mon, 02 Oct 2023 04:17:28 UTC

Yes. OpenObserve flattens the structure with or without VRL

Photo of Mik
Mik
Mon, 02 Oct 2023 04:17:54 UTC

Thanks Prabhat

Photo of Mik
Mik
Mon, 02 Oct 2023 04:32:05 UTC

Prabhat just some inquiries: 1. May I know why? 2. Is this a behaviour we can disable? 3. Is this intended to be the case in the future? Thank you in advance! :pray:

Photo of Prabhat
Prabhat
Mon, 02 Oct 2023 04:34:50 UTC

data is stored in tabular format in parquet files and flattening is the best option. you can't disable it. it won't change . We understand that there can be inconveniences with this approach, but we think its a good tradeoff.

Photo of Mik
Mik
Mon, 02 Oct 2023 04:36:13 UTC

Thanks for the clarification. Appreciate it!