Transforming Message Values for Filtering
TLDR arun sought advice on value transformation for filtering and received guidance from Prabhat to create a VRL function. After troubleshooting, arun successfully implemented the transform rule in vector.toml.
Prabhat
Fri, 01 Sep 2023 11:16:46 UTCYou coul create a VRL function with the following contents and apply it to your stream during ingestion for your above data: ```.message = parse_json!(.message) .level = .message.level del(.message.level) .```
Prabhat
Fri, 01 Sep 2023 11:25:16 UTCtry
arun
Fri, 01 Sep 2023 11:34:48 UTCohh a playground, nice! for some reason im not seeing them split up yet. some of my docker containers do not have the message key as json always. like nginx will just have a string but our inhouse modules will always output message as json
arun
Fri, 01 Sep 2023 11:35:02 UTCso i also get errors like ERROR transform{component_kind="transform" component_id=remap_parse component_type=remap component_name=remap_parse}: vector::internal_events::remap: Mapping failed with event. error="function call error for \"parse_json\" at (12:33): unable to parse json: expected `,` or `]` at line 1 column 6" error_type="conversion_failed" stage="processing" internal_log_rate_limit=true
arun
Fri, 01 Sep 2023 11:35:55 UTC```[transforms.remap_parse] type = "remap" inputs = ["my_docker_logs_source"] source = ''' .message = parse_json!(.message) .level = .message.level del(.message.level) . ''' ```
Prabhat
Fri, 01 Sep 2023 11:39:43 UTCtry ```.message, error = parse_json(.message) if error == null { .level = .message.level del(.message.level) } else { del(.message) } .```
Prabhat
Fri, 01 Sep 2023 11:39:56 UTCYou can handle such errors in VRL
Prabhat
Fri, 01 Sep 2023 11:40:18 UTCWhile you can do these in vector.toml as well
Prabhat
Fri, 01 Sep 2023 11:40:28 UTCYou can do these in OpenObserve UI too
arun
Fri, 01 Sep 2023 11:47:27 UTCOh i see, i have not noticed that somehow. when i paste the most recent VRL function in the OO UI it works. but for some reason vector does not seem to work
arun
Fri, 01 Sep 2023 11:47:44 UTCi mean vector container is working as it's pushing logs.
Prabhat
Fri, 01 Sep 2023 11:49:07 UTCYou will need to associate it with the stream. Try -
arun
Fri, 01 Sep 2023 11:54:53 UTCIs it possible to have this in vector.toml? I am trying to make the combo openobserve + vector to be deployable via IaC
Prabhat
Fri, 01 Sep 2023 11:56:32 UTCyes, you could
Prabhat
Fri, 01 Sep 2023 11:56:59 UTCjust update your script in vector.toml
arun
Fri, 01 Sep 2023 11:57:27 UTC```[transforms.remap_transform] type = "remap" inputs = ["my_docker_logs_source"] source = ''' .message, error = parse_json(.message) if error == null { .level = .message.level del(.message.level) } else { del(.message) } . ''' ``` This is my transform section but it does not seem to work (reloaded vector container)
Prabhat
Fri, 01 Sep 2023 11:58:41 UTCtry changing it to : ```[transforms.remap_transform] type = "remap" inputs = ["my_docker_logs_source"] source = ''' .message, error = parse_json(.message) if error == null { .level = .message.level del(.message.level) } else { del(.message) } ''' ```
arun
Fri, 01 Sep 2023 11:58:56 UTCohh, was it the dot at the end?
Prabhat
Fri, 01 Sep 2023 11:59:04 UTCremoved "." dot at the end that is needed in OpenObserve UI
arun
Fri, 01 Sep 2023 12:04:42 UTCthanks, i've removed that, reloaded vector and openobserve. the messages are still not transformed. double checked and copy pasted your snippet as well
arun
Fri, 01 Sep 2023 12:05:11 UTCin the vector logs i only see occasional "[Received out of order log message.]" messages which i got before as well
arun
Fri, 01 Sep 2023 12:06:26 UTCif it matter,s this is the configured source ```[sources.my_docker_logs_source] type = "docker_logs" auto_partial_merge = true exclude_containers = [ "install" ] retry_backoff_secs = 2 ```
arun
Fri, 01 Sep 2023 12:13:19 UTChmm, i tried the "validate" option from vector CLI, ```root@c159e9f1d209:/# vector validate /etc/vector/vecto Loaded with warnings ["/etc/vector/vector.toml"] ------------------------------------------------ ~ Transform "remap_transform" has no consumers```
arun
Fri, 01 Sep 2023 12:17:05 UTCi think i solved it, i had to put `remap_transform` into the input of my sink
arun
Fri, 01 Sep 2023 12:17:15 UTCill wait a bit to get some more data
OpenObserve
Indexed 419 threads
OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI
View in App
arun
Fri, 01 Sep 2023 08:14:46 UTCHey, not so openobserve related but i want to transform one of the values to further filter ```{ "_timestamp": 1693555345288389, "container_created_at": "2023-08-31T07:28:40.193751899Z", "container_id": "9e26cfb05be8cff76d23453453453459abf810b3f6e30fe1289", "container_name": "digpla.1.sq2mdewerwerwrj7o5sgbfwo", "host": "5a189794d70e", "image": "image.som:65@sha256:60de2a8e534534534534569b968940fda1f6866", "label_com_docker_stack_namespace": "digital_audit", "label_com_docker_swarm_node_id": "2ni9isg1bm7fbtoqd45e15z6k", "label_com_docker_swarm_service_id": "s48396x7i33gculgjy7b9szrt", "label_com_docker_swarm_service_name": "digpla", "label_com_docker_swarm_task": "", "label_com_docker_swarm_task_id": "sq2md6hdpnvshcrj7o5sgbfwo", "label_com_docker_swarm_task_name": "digpla.1.sq2mdewerwerwrj7o5sgbfwo", "message": "{\"@timestamp\":\"2023-09-01T08:06:18.437+00:00\",\"thread_name\":\"SimpleAsyncTaskExecutor-2\",\"logger_name\":\"asdszgdfzgergdgdf\",\"level\":\"DEBUG\",\"message\":\"Indexing: file=/var/data/dfgdsfgdlog, offset=13590\",\"details\":\"\"}", "source_type": "docker_logs", "stream": "stdout", "timestamp": "2023-09-01T08:06:18.437471039Z" }``` I have the `message` value that is a json and i want to filter that automatically. ELK seems to be able to detect the timestamp automatically when creating an index pattern and also seems to offer automatically the `level` keyword for example which is very useful when filtering. I use Vector to ingest the values but im not so familiar with its transform rules