TLDR Ivan encountered an issue while setting up Kafka logs with OpenObserve and shared their vector config. Prabhat suggested a possible resource issue, but a final resolution wasn't determined.
Vector has access to openobserve
Are both vector and OpenObserve on same machines? Even though OpenObserve does not require a lot of CPU or memory it might not be getting enough of it.
Do you think that could be the case?
Ivan
Wed, 19 Jul 2023 14:48:32 UTCHi, can you help me setup the logs that I want to take from kafka and put in openobseve my vector config ```vector.yaml: | api: address: 0.0.0.0:8686 enabled: true playground: false healthchecks: enabled: true require_healthy: true sinks: openobserve: batch: max_bytes: 10000 timeout_secs: 5 buffer: max_events: 70 auth: password: 2BtQvhXlGqwEsl1Z strategy: basic user:
compression: gzip
encoding:
codec: json
timestamp_format: rfc3339
healthcheck:
enabled: false
inputs:
- throttle_out
method: post
type: http
uri:
prometheus:
address: 0.0.0.0:9598
default_namespace: service
inputs:
- internal_metrics
type: prometheus_exporter
sources:
internal_metrics:
namespace: vector
scrape_interval_secs: 30
type: internal_metrics
msk:
acknowledgements:
enabled: true
bootstrap_servers: $KAFKA_BROKERS
decoding:
codec: json
group_id: vector-openobserve
librdkafka_options:
queued.max.messages.kbytes: "101920"
queued.min.messages: "10"
sasl:
enabled: true
mechanism: SCRAM-SHA-512
password: $KAFKA_PASSWORD
username: $KAFKA_USERNAME
tls:
ca_file: /secrets/ca.cert
enabled: true
topics:
- logs.test.test
type: kafka
transforms:
create_index:
drop_on_abort: true
drop_on_error: true
inputs:
- throttle_in
source: |-
del(.headers)
if exists(.kubernetes.pod_labels."pod-template-hash") {
.deployment_with_hash, err = replace(.kubernetes.pod_name, .kubernetes.pod_labels."pod-template-hash", "", count: 1)
.deployment = replace(.deployment_with_hash, r'--[^-]+$', "", count: 1)
del(.deployment_with_hash)
} else {
.deployment, err = replace(.kubernetes.pod_name, r'-[^-]+$', "", count: 1)
}
if exists(.kubernetes.pod_labels.stream) {
.stream, err = downcase(.kubernetes.pod_labels.stream)
if err != null {
log("Downcase error: " + err, level: "error", rate_limit_secs: 60)
}
} else {
.stream = "unallocated"
}
if !exists(."@timestamp") {
."@timestamp" = .ingestion_ts
}
type: remap
throttle_in:
inputs:
- msk
threshold: 1500
type: throttle
window_secs: 1
throttle_out:
inputs:
- create_index
key_field: '{{ stream }}'
threshold: 500
type: throttle
window_secs: 1```
I got such error
```WARN sink{component_kind="sink" component_id=openobserve component_type=http component_name=openobserve}:request{request_id=0}: vector::sinks::util::retries: Request timed out. If this happens often while the events are actually reaching their destination, try decreasing `batch.max_bytes` and/or using `compression` if applicable. Alternatively `request.timeout_secs` can be increased. internal_log_rate_limit=true```
Thanks