Transforming Message Values for Filtering

TLDR arun sought advice on value transformation for filtering and received guidance from Prabhat to create a VRL function. After troubleshooting, arun successfully implemented the transform rule in vector.toml.

Photo of arun
arun
Fri, 01 Sep 2023 08:14:46 UTC

Hey, not so openobserve related but i want to transform one of the values to further filter ```{ "_timestamp": 1693555345288389, "container_created_at": "2023-08-31T07:28:40.193751899Z", "container_id": "9e26cfb05be8cff76d23453453453459abf810b3f6e30fe1289", "container_name": "digpla.1.sq2mdewerwerwrj7o5sgbfwo", "host": "5a189794d70e", "image": "image.som:65@sha256:60de2a8e534534534534569b968940fda1f6866", "label_com_docker_stack_namespace": "digital_audit", "label_com_docker_swarm_node_id": "2ni9isg1bm7fbtoqd45e15z6k", "label_com_docker_swarm_service_id": "s48396x7i33gculgjy7b9szrt", "label_com_docker_swarm_service_name": "digpla", "label_com_docker_swarm_task": "", "label_com_docker_swarm_task_id": "sq2md6hdpnvshcrj7o5sgbfwo", "label_com_docker_swarm_task_name": "digpla.1.sq2mdewerwerwrj7o5sgbfwo", "message": "{\"@timestamp\":\"2023-09-01T08:06:18.437+00:00\",\"thread_name\":\"SimpleAsyncTaskExecutor-2\",\"logger_name\":\"asdszgdfzgergdgdf\",\"level\":\"DEBUG\",\"message\":\"Indexing: file=/var/data/dfgdsfgdlog, offset=13590\",\"details\":\"\"}", "source_type": "docker_logs", "stream": "stdout", "timestamp": "2023-09-01T08:06:18.437471039Z" }``` I have the `message` value that is a json and i want to filter that automatically. ELK seems to be able to detect the timestamp automatically when creating an index pattern and also seems to offer automatically the `level` keyword for example which is very useful when filtering. I use Vector to ingest the values but im not so familiar with its transform rules

Photo of Prabhat
Prabhat
Fri, 01 Sep 2023 11:16:46 UTC

You coul create a VRL function with the following contents and apply it to your stream during ingestion for your above data: ```.message = parse_json!(.message) .level = .message.level del(.message.level) .```

Photo of Prabhat
Prabhat
Fri, 01 Sep 2023 11:25:16 UTC

try

Photo of arun
arun
Fri, 01 Sep 2023 11:34:48 UTC

ohh a playground, nice! for some reason im not seeing them split up yet. some of my docker containers do not have the message key as json always. like nginx will just have a string but our inhouse modules will always output message as json

Photo of arun
arun
Fri, 01 Sep 2023 11:35:02 UTC

so i also get errors like ERROR transform{component_kind="transform" component_id=remap_parse component_type=remap component_name=remap_parse}: vector::internal_events::remap: Mapping failed with event. error="function call error for \"parse_json\" at (12:33): unable to parse json: expected `,` or `]` at line 1 column 6" error_type="conversion_failed" stage="processing" internal_log_rate_limit=true

Photo of arun
arun
Fri, 01 Sep 2023 11:35:55 UTC

```[transforms.remap_parse] type = "remap" inputs = ["my_docker_logs_source"] source = ''' .message = parse_json!(.message) .level = .message.level del(.message.level) . ''' ```

Photo of Prabhat
Prabhat
Fri, 01 Sep 2023 11:39:43 UTC

try ```.message, error = parse_json(.message) if error == null { .level = .message.level del(.message.level) } else { del(.message) } .```

Photo of Prabhat
Prabhat
Fri, 01 Sep 2023 11:39:56 UTC

You can handle such errors in VRL

Photo of Prabhat
Prabhat
Fri, 01 Sep 2023 11:40:18 UTC

While you can do these in vector.toml as well

Photo of Prabhat
Prabhat
Fri, 01 Sep 2023 11:40:28 UTC

You can do these in OpenObserve UI too

Photo of arun
arun
Fri, 01 Sep 2023 11:47:27 UTC

Oh i see, i have not noticed that somehow. when i paste the most recent VRL function in the OO UI it works. but for some reason vector does not seem to work

Photo of arun
arun
Fri, 01 Sep 2023 11:47:44 UTC

i mean vector container is working as it's pushing logs.

Photo of Prabhat
Prabhat
Fri, 01 Sep 2023 11:49:07 UTC

You will need to associate it with the stream. Try -

Photo of arun
arun
Fri, 01 Sep 2023 11:54:53 UTC

Is it possible to have this in vector.toml? I am trying to make the combo openobserve + vector to be deployable via IaC

Photo of Prabhat
Prabhat
Fri, 01 Sep 2023 11:56:32 UTC

yes, you could

Photo of Prabhat
Prabhat
Fri, 01 Sep 2023 11:56:59 UTC

just update your script in vector.toml

Photo of arun
arun
Fri, 01 Sep 2023 11:57:27 UTC

```[transforms.remap_transform] type = "remap" inputs = ["my_docker_logs_source"] source = ''' .message, error = parse_json(.message) if error == null { .level = .message.level del(.message.level) } else { del(.message) } . ''' ``` This is my transform section but it does not seem to work (reloaded vector container)

Photo of Prabhat
Prabhat
Fri, 01 Sep 2023 11:58:41 UTC

try changing it to : ```[transforms.remap_transform] type = "remap" inputs = ["my_docker_logs_source"] source = ''' .message, error = parse_json(.message) if error == null { .level = .message.level del(.message.level) } else { del(.message) } ''' ```

Photo of arun
arun
Fri, 01 Sep 2023 11:58:56 UTC

ohh, was it the dot at the end?

Photo of Prabhat
Prabhat
Fri, 01 Sep 2023 11:59:04 UTC

removed "." dot at the end that is needed in OpenObserve UI

Photo of arun
arun
Fri, 01 Sep 2023 12:04:42 UTC

thanks, i've removed that, reloaded vector and openobserve. the messages are still not transformed. double checked and copy pasted your snippet as well

Photo of arun
arun
Fri, 01 Sep 2023 12:05:11 UTC

in the vector logs i only see occasional "[Received out of order log message.]" messages which i got before as well

Photo of arun
arun
Fri, 01 Sep 2023 12:06:26 UTC

if it matter,s this is the configured source ```[sources.my_docker_logs_source] type = "docker_logs" auto_partial_merge = true exclude_containers = [ "install" ] retry_backoff_secs = 2 ```

Photo of arun
arun
Fri, 01 Sep 2023 12:13:19 UTC

hmm, i tried the "validate" option from vector CLI, ```root@c159e9f1d209:/# vector validate /etc/vector/vecto Loaded with warnings ["/etc/vector/vector.toml"] ------------------------------------------------ ~ Transform "remap_transform" has no consumers```

Photo of arun
arun
Fri, 01 Sep 2023 12:17:05 UTC

i think i solved it, i had to put `remap_transform` into the input of my sink

Photo of arun
arun
Fri, 01 Sep 2023 12:17:15 UTC

ill wait a bit to get some more data