Inner Workings of OpenObserve VRL Function
TLDR Mik inquired about the VRL function behavior in OpenObserve logs. Prabhat explained the system automatically flattens structures, a feature that won't change in the future.
Prabhat
Mon, 02 Oct 2023 04:17:28 UTCYes. OpenObserve flattens the structure with or without VRL
Mik
Mon, 02 Oct 2023 04:17:54 UTCThanks Prabhat
Mik
Mon, 02 Oct 2023 04:32:05 UTCPrabhat just some inquiries: 1. May I know why? 2. Is this a behaviour we can disable? 3. Is this intended to be the case in the future? Thank you in advance! :pray:
Prabhat
Mon, 02 Oct 2023 04:34:50 UTCdata is stored in tabular format in parquet files and flattening is the best option. you can't disable it. it won't change . We understand that there can be inconveniences with this approach, but we think its a good tradeoff.
Mik
Mon, 02 Oct 2023 04:36:13 UTCThanks for the clarification. Appreciate it!
OpenObserve
Indexed 419 threads
OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI
View in App
Mik
Mon, 02 Oct 2023 04:15:56 UTCHi everyone! Is this the expected behavior of the VRL function in the open observe logs? VRL function ```.log = parse_json!(.message_message) del(.message_message) .``` VRL playground output: ```{ "_timestamp": 1695956833809000, "id": "37821101218481254744296987650977054435587665849865928704", "log": { "applicationName": "sbc-outbound", "event": { "message": "Refresh RTP Engine connections: 10.100.12.93" }, "eventTime": 1695956829, "instanceName": "sip-little-woodpecker", "level": "info" }, "loggroup": "/log/sbc-sip/sbc-log", "logstream": "sip-little-woodpecker", ... "requestid": "11111xxxx-fe77-481c-b60b-73a33a1d9602", "subscriptionfilters": "[\"CloudwatchOpenobserveIngestStack-SubscriptionFilterXXXXX-wZGu0KhDhTIC\"]", "timestamp": 1695956833809 }``` OpenObserve logs: ```{ _timestamp:1695956113809000, id:37821085161944711802248324874853977452967836770446606336, log_applicationname:sbc-outbound, log_event_message:Refresh RTP Engine connections: 10.100.12.93, log_eventtime:1695956109, log_instancename:sip-little-woodpecker, log_level:info, "loggroup": "/log/sbc-sip/sbc-log", "logstream": "sip-little-woodpecker", ... "requestid": "11111xxxx-fe77-481c-b60b-73a33a1d9602", "subscriptionfilters": "[\"CloudwatchOpenobserveIngestStack-SubscriptionFilterXXXXX-wZGu0KhDhTIC\"]", "timestamp": 1695956833809 }``` OpenObserve will automatically append an `underscore` for nested JSON object keys?