Troubleshooting Kafka Log Setup with OpenObserve

TLDR Ivan encountered an issue while setting up Kafka logs with OpenObserve and shared their vector config. Prabhat suggested a possible resource issue, but a final resolution wasn't determined.

Photo of Ivan
Ivan
Wed, 19 Jul 2023 14:48:32 UTC

Hi, can you help me setup the logs that I want to take from kafka and put in openobseve my vector config ```vector.yaml: | api: address: 0.0.0.0:8686 enabled: true playground: false healthchecks: enabled: true require_healthy: true sinks: openobserve: batch: max_bytes: 10000 timeout_secs: 5 buffer: max_events: 70 auth: password: 2BtQvhXlGqwEsl1Z strategy: basic user: compression: gzip encoding: codec: json timestamp_format: rfc3339 healthcheck: enabled: false inputs: - throttle_out method: post type: http uri: prometheus: address: 0.0.0.0:9598 default_namespace: service inputs: - internal_metrics type: prometheus_exporter sources: internal_metrics: namespace: vector scrape_interval_secs: 30 type: internal_metrics msk: acknowledgements: enabled: true bootstrap_servers: $KAFKA_BROKERS decoding: codec: json group_id: vector-openobserve librdkafka_options: queued.max.messages.kbytes: "101920" queued.min.messages: "10" sasl: enabled: true mechanism: SCRAM-SHA-512 password: $KAFKA_PASSWORD username: $KAFKA_USERNAME tls: ca_file: /secrets/ca.cert enabled: true topics: - logs.test.test type: kafka transforms: create_index: drop_on_abort: true drop_on_error: true inputs: - throttle_in source: |- del(.headers) if exists(.kubernetes.pod_labels."pod-template-hash") { .deployment_with_hash, err = replace(.kubernetes.pod_name, .kubernetes.pod_labels."pod-template-hash", "", count: 1) .deployment = replace(.deployment_with_hash, r'--[^-]+$', "", count: 1) del(.deployment_with_hash) } else { .deployment, err = replace(.kubernetes.pod_name, r'-[^-]+$', "", count: 1) } if exists(.kubernetes.pod_labels.stream) { .stream, err = downcase(.kubernetes.pod_labels.stream) if err != null { log("Downcase error: " + err, level: "error", rate_limit_secs: 60) } } else { .stream = "unallocated" } if !exists(."@timestamp") { ."@timestamp" = .ingestion_ts } type: remap throttle_in: inputs: - msk threshold: 1500 type: throttle window_secs: 1 throttle_out: inputs: - create_index key_field: '{{ stream }}' threshold: 500 type: throttle window_secs: 1``` I got such error ```WARN sink{component_kind="sink" component_id=openobserve component_type=http component_name=openobserve}:request{request_id=0}: vector::sinks::util::retries: Request timed out. If this happens often while the events are actually reaching their destination, try decreasing `batch.max_bytes` and/or using `compression` if applicable. Alternatively `request.timeout_secs` can be increased. internal_log_rate_limit=true``` Thanks

Photo of Ivan
Ivan
Wed, 19 Jul 2023 14:50:28 UTC

Vector has access to openobserve

Photo of Prabhat
Prabhat
Thu, 20 Jul 2023 17:54:22 UTC

Are both vector and OpenObserve on same machines? Even though OpenObserve does not require a lot of CPU or memory it might not be getting enough of it.

Photo of Prabhat
Prabhat
Thu, 20 Jul 2023 17:54:37 UTC

Do you think that could be the case?