TLDR arun sought advice on value transformation for filtering and received guidance from Prabhat to create a VRL function. After troubleshooting, arun successfully implemented the transform rule in vector.toml.
You coul create a VRL function with the following contents and apply it to your stream during ingestion for your above data: ```.message = parse_json!(.message) .level = .message.level del(.message.level) .```
try
ohh a playground, nice! for some reason im not seeing them split up yet. some of my docker containers do not have the message key as json always. like nginx will just have a string but our inhouse modules will always output message as json
so i also get errors like ERROR transform{component_kind="transform" component_id=remap_parse component_type=remap component_name=remap_parse}: vector::internal_events::remap: Mapping failed with event. error="function call error for \"parse_json\" at (12:33): unable to parse json: expected `,` or `]` at line 1 column 6" error_type="conversion_failed" stage="processing" internal_log_rate_limit=true
```[transforms.remap_parse] type = "remap" inputs = ["my_docker_logs_source"] source = ''' .message = parse_json!(.message) .level = .message.level del(.message.level) . ''' ```
try ```.message, error = parse_json(.message) if error == null { .level = .message.level del(.message.level) } else { del(.message) } .```
You can handle such errors in VRL
While you can do these in vector.toml as well
You can do these in OpenObserve UI too
Oh i see, i have not noticed that somehow. when i paste the most recent VRL function in the OO UI it works. but for some reason vector does not seem to work
i mean vector container is working as it's pushing logs.
You will need to associate it with the stream. Try -
Is it possible to have this in vector.toml? I am trying to make the combo openobserve + vector to be deployable via IaC
yes, you could
just update your script in vector.toml
```[transforms.remap_transform] type = "remap" inputs = ["my_docker_logs_source"] source = ''' .message, error = parse_json(.message) if error == null { .level = .message.level del(.message.level) } else { del(.message) } . ''' ``` This is my transform section but it does not seem to work (reloaded vector container)
try changing it to : ```[transforms.remap_transform] type = "remap" inputs = ["my_docker_logs_source"] source = ''' .message, error = parse_json(.message) if error == null { .level = .message.level del(.message.level) } else { del(.message) } ''' ```
ohh, was it the dot at the end?
removed "." dot at the end that is needed in OpenObserve UI
thanks, i've removed that, reloaded vector and openobserve. the messages are still not transformed. double checked and copy pasted your snippet as well
in the vector logs i only see occasional "[Received out of order log message.]" messages which i got before as well
if it matter,s this is the configured source ```[sources.my_docker_logs_source] type = "docker_logs" auto_partial_merge = true exclude_containers = [ "install" ] retry_backoff_secs = 2 ```
hmm, i tried the "validate" option from vector CLI, ```root@c159e9f1d209:/# vector validate /etc/vector/vecto Loaded with warnings ["/etc/vector/vector.toml"] ------------------------------------------------ ~ Transform "remap_transform" has no consumers```
i think i solved it, i had to put `remap_transform` into the input of my sink
ill wait a bit to get some more data
arun
Fri, 01 Sep 2023 08:14:46 UTCHey, not so openobserve related but i want to transform one of the values to further filter ```{ "_timestamp": 1693555345288389, "container_created_at": "2023-08-31T07:28:40.193751899Z", "container_id": "9e26cfb05be8cff76d23453453453459abf810b3f6e30fe1289", "container_name": "digpla.1.sq2mdewerwerwrj7o5sgbfwo", "host": "5a189794d70e", "image": "image.som:65@sha256:60de2a8e534534534534569b968940fda1f6866", "label_com_docker_stack_namespace": "digital_audit", "label_com_docker_swarm_node_id": "2ni9isg1bm7fbtoqd45e15z6k", "label_com_docker_swarm_service_id": "s48396x7i33gculgjy7b9szrt", "label_com_docker_swarm_service_name": "digpla", "label_com_docker_swarm_task": "", "label_com_docker_swarm_task_id": "sq2md6hdpnvshcrj7o5sgbfwo", "label_com_docker_swarm_task_name": "digpla.1.sq2mdewerwerwrj7o5sgbfwo", "message": "{\"@timestamp\":\"2023-09-01T08:06:18.437+00:00\",\"thread_name\":\"SimpleAsyncTaskExecutor-2\",\"logger_name\":\"asdszgdfzgergdgdf\",\"level\":\"DEBUG\",\"message\":\"Indexing: file=/var/data/dfgdsfgdlog, offset=13590\",\"details\":\"\"}", "source_type": "docker_logs", "stream": "stdout", "timestamp": "2023-09-01T08:06:18.437471039Z" }``` I have the `message` value that is a json and i want to filter that automatically. ELK seems to be able to detect the timestamp automatically when creating an index pattern and also seems to offer automatically the `level` keyword for example which is very useful when filtering. I use Vector to ingest the values but im not so familiar with its transform rules