Issues with Vector Log Transmission to OpenObserve

TLDR Ivan was having trouble setting a dynamic sink url in Vector for OpenObserve. Chris suggested that he needed a separate vector sink for each org/stream as current Vector version didn't support dynamic construction of sink url.

Photo of Ivan
Ivan
Thu, 17 Aug 2023 10:25:52 UTC

Hi, need advice, maybe this problem has already been solved, I am using vector to send logs to openobserve Config vector: ```sinks: openobserve: type: http inputs: - throttle_out uri: '{{"{{url}}"}}' method: post auth: strategy: basic user: "" password: "Complexpass#123" compression: gzip encoding: codec: json timestamp_format: rfc3339 healthcheck: enabled: false transforms: throttle_in: inputs: - msk threshold: 1500 type: throttle window_secs: 1 throttle_out: inputs: - create_index threshold: 500 key_field: '{{ "{{" }} stream {{ "}}" }}' type: throttle window_secs: 1 create_index: drop_on_abort: true drop_on_error: true inputs: - throttle_in source: |- if exists(.) { .app, err = downcase(.) if err != null { log("Downcase error: " + err, level: "error", rate_limit_secs: 60) } } else { .app = "unallocated" } .url = "" + .kubernetes.pod_namespace + "/" + .app + "/_json"``` I want to separate logs by organization=namespace in kubernetes and stream=service(app) Well, the problem is that vector does not accept the value of a variable in uri: '{{"{{url}}"}}' Maybe you somehow solved this problem by separating the organization and stream through variables in the vector Thanks

Photo of Prabhat
Prabhat
Thu, 17 Aug 2023 10:34:02 UTC

Haven't dealt with this yet. I checked the config and could not find something quickly. vector discord server has folks who will be more knowdgeable and should be able to help.

Photo of Chris
Chris
Thu, 17 Aug 2023 10:46:53 UTC

When i looked a few weeks ago, vector doesn't support dynamic construction of the sink url. It is on their wish-list (can't remember whether they have committed to implementing it). But at the moment you need a separate vector sink for each org/stream

Photo of Ivan
Ivan
Thu, 17 Aug 2023 10:48:38 UTC

Got it, thanks for the replies

Photo of Chris
Chris
Thu, 17 Aug 2023 10:51:57 UTC

i see vector-0.32.0 was released on 15 Aug... at a quick glance i don't see anything about this but haven't looked in detail yet