TLDR Jennings shared a solution they made for using OpenObserve and Vector on Kubernetes. A solution to Vector's service discovery challenge was discussed, and West shared their custom OTEL collector exporter. They concluded the discussion with Prabhat sharing a useful configuration link, and West sharing useful yaml templates.
Jennings Thank you for doing this. Vector is great for logs. One of its challenges when it comes to prometheus metrics is that it does not support service discovery. That would eb a deal breaker in most of the cases.
We are working on an otel-collector based system that will help capture logs and metrics. It will also help auto-instrument traces for most used programming languages. You can use it today -
I will be sure to try it out!
Do you know if there is a better way to ship kubelet `/metrics/cadvisor` metrics to OpenObserve? I am not sure how one would do that with otel
OpenObserve-collector is configured to send advisor metrics to OpenObserve. Otel-collector can scrape it directly. There is also kubeletstats receiver that does most of it.
Prabhat, We have created custom OTEL collector exporter to ship logs to openObserve (we are shipping logs , k8s events) , also we are using OTEL prometheus receiver to scrape metrics and prometheus remote write exporter to export metrics to Openobserve , We are using OTLP exporter to export traces . We can cover logs, metrics, and traces through OTEL collector
Jennings , Pls refer this OTEL configuration , it may help you .
prometheus:
config:
global:
scrape_interval: 150s
scrape_timeout: 10s
scrape_configs:
- job_name: blackbox-prometheus-blackbox-exporter
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets:
-
Cool. Official collector does the same thing. It uses otel-collector operator to simplify things.
Prabhat I will take a look. West thanks for sharing your YAMLs
Jennings
Mon, 16 Oct 2023 20:03:46 UTCHello, I wanted to share something I made! I have a canned solution for using OpenObserve and on Kubernetes. It's a 2 piece, 100% Rust observability stack which is very lightweight compared to the status quo (Prometheus, Thanos, Grafana, Loki).
How it works is pretty simple, but it does require a helper script to configure Vector to collect metrics from Kubelet. I've created an issue under Vector to discuss ways this can be improved. The issue and how to run -->