Troubleshooting Kafka Log Setup with OpenObserve
TLDR Ivan encountered an issue while setting up Kafka logs with OpenObserve and shared their vector config. Prabhat suggested a possible resource issue, but a final resolution wasn't determined.
Ivan
Wed, 19 Jul 2023 14:50:28 UTCVector has access to openobserve
Prabhat
Thu, 20 Jul 2023 17:54:22 UTCAre both vector and OpenObserve on same machines? Even though OpenObserve does not require a lot of CPU or memory it might not be getting enough of it.
Prabhat
Thu, 20 Jul 2023 17:54:37 UTCDo you think that could be the case?
OpenObserve
Indexed 419 threads
OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI
View in App
Ivan
Wed, 19 Jul 2023 14:48:32 UTCHi, can you help me setup the logs that I want to take from kafka and put in openobseve my vector config ```vector.yaml: | api: address: 0.0.0.0:8686 enabled: true playground: false healthchecks: enabled: true require_healthy: true sinks: openobserve: batch: max_bytes: 10000 timeout_secs: 5 buffer: max_events: 70 auth: password: 2BtQvhXlGqwEsl1Z strategy: basic user:
compression: gzip
encoding:
codec: json
timestamp_format: rfc3339
healthcheck:
enabled: false
inputs:
- throttle_out
method: post
type: http
uri:
prometheus:
address: 0.0.0.0:9598
default_namespace: service
inputs:
- internal_metrics
type: prometheus_exporter
sources:
internal_metrics:
namespace: vector
scrape_interval_secs: 30
type: internal_metrics
msk:
acknowledgements:
enabled: true
bootstrap_servers: $KAFKA_BROKERS
decoding:
codec: json
group_id: vector-openobserve
librdkafka_options:
queued.max.messages.kbytes: "101920"
queued.min.messages: "10"
sasl:
enabled: true
mechanism: SCRAM-SHA-512
password: $KAFKA_PASSWORD
username: $KAFKA_USERNAME
tls:
ca_file: /secrets/ca.cert
enabled: true
topics:
- logs.test.test
type: kafka
transforms:
create_index:
drop_on_abort: true
drop_on_error: true
inputs:
- throttle_in
source: |-
del(.headers)
if exists(.kubernetes.pod_labels."pod-template-hash") {
.deployment_with_hash, err = replace(.kubernetes.pod_name, .kubernetes.pod_labels."pod-template-hash", "", count: 1)
.deployment = replace(.deployment_with_hash, r'--[^-]+$', "", count: 1)
del(.deployment_with_hash)
} else {
.deployment, err = replace(.kubernetes.pod_name, r'-[^-]+$', "", count: 1)
}
if exists(.kubernetes.pod_labels.stream) {
.stream, err = downcase(.kubernetes.pod_labels.stream)
if err != null {
log("Downcase error: " + err, level: "error", rate_limit_secs: 60)
}
} else {
.stream = "unallocated"
}
if !exists(."@timestamp") {
."@timestamp" = .ingestion_ts
}
type: remap
throttle_in:
inputs:
- msk
threshold: 1500
type: throttle
window_secs: 1
throttle_out:
inputs:
- create_index
key_field: '{{ stream }}'
threshold: 500
type: throttle
window_secs: 1```
I got such error
```WARN sink{component_kind="sink" component_id=openobserve component_type=http component_name=openobserve}:request{request_id=0}: vector::sinks::util::retries: Request timed out. If this happens often while the events are actually reaching their destination, try decreasing `batch.max_bytes` and/or using `compression` if applicable. Alternatively `request.timeout_secs` can be increased. internal_log_rate_limit=true```
Thanks