Discussing OpenObserve, Vector and Prometheus Metrics

TLDR Jennings shared a solution they made for using OpenObserve and Vector on Kubernetes. A solution to Vector's service discovery challenge was discussed, and West shared their custom OTEL collector exporter. They concluded the discussion with Prabhat sharing a useful configuration link, and West sharing useful yaml templates.

Photo of Jennings
Jennings
Mon, 16 Oct 2023 20:03:46 UTC

Hello, I wanted to share something I made! I have a canned solution for using OpenObserve and on Kubernetes. It's a 2 piece, 100% Rust observability stack which is very lightweight compared to the status quo (Prometheus, Thanos, Grafana, Loki). How it works is pretty simple, but it does require a helper script to configure Vector to collect metrics from Kubelet. I've created an issue under Vector to discuss ways this can be improved. The issue and how to run -->

Photo of Prabhat
Prabhat
Mon, 16 Oct 2023 21:23:27 UTC

Jennings Thank you for doing this. Vector is great for logs. One of its challenges when it comes to prometheus metrics is that it does not support service discovery. That would eb a deal breaker in most of the cases.

Photo of Prabhat
Prabhat
Mon, 16 Oct 2023 21:26:13 UTC

We are working on an otel-collector based system that will help capture logs and metrics. It will also help auto-instrument traces for most used programming languages. You can use it today -

Photo of Jennings
Jennings
Mon, 16 Oct 2023 23:28:38 UTC

I will be sure to try it out!

Photo of Jennings
Jennings
Mon, 16 Oct 2023 23:33:02 UTC

Do you know if there is a better way to ship kubelet `/metrics/cadvisor` metrics to OpenObserve? I am not sure how one would do that with otel

Photo of Prabhat
Prabhat
Tue, 17 Oct 2023 00:08:46 UTC

OpenObserve-collector is configured to send advisor metrics to OpenObserve. Otel-collector can scrape it directly. There is also kubeletstats receiver that does most of it.

Photo of West
West
Tue, 17 Oct 2023 05:22:17 UTC

Prabhat, We have created custom OTEL collector exporter to ship logs to openObserve (we are shipping logs , k8s events) , also we are using OTEL prometheus receiver to scrape metrics and prometheus remote write exporter to export metrics to Openobserve , We are using OTLP exporter to export traces . We can cover logs, metrics, and traces through OTEL collector

Photo of West
West
Tue, 17 Oct 2023 05:26:29 UTC

Jennings , Pls refer this OTEL configuration , it may help you . prometheus: config: global: scrape_interval: 150s scrape_timeout: 10s scrape_configs: - job_name: blackbox-prometheus-blackbox-exporter metrics_path: /probe params: module: [http_2xx] static_configs: - targets: - relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox.prometheus.svc.cluster.local:9115 - job_name: internal-service-monitoring kubernetes_sd_configs: - role: service metrics_path: /probe params: module: - http_2xx relabel_configs: - action: keep regex: true source_labels: - __meta_kubernetes_service_annotation_prometheus_io_probe - source_labels: - __address__ target_label: __param_target - replacement: blackbox.prometheus.svc.cluster.local:9115 target_label: __address__ - source_labels: - __param_target target_label: instance - action: labelmap regex: __meta_kubernetes_service_label_(.+) - source_labels: - __meta_kubernetes_namespace target_label: kubernetes_namespace - source_labels: - __meta_kubernetes_service_name target_label: kubernetes_name - job_name: kubernetes-apiservers bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: endpoints relabel_configs: - action: keep regex: default;kubernetes;https source_labels: - __meta_kubernetes_namespace - __meta_kubernetes_service_name - __meta_kubernetes_endpoint_port_name scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true - job_name: kubernetes-nodes bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: node relabel_configs: - action: labelmap regex: __meta_kubernetes_node_label_(.+) - replacement: kubernetes.default.svc:443 target_label: __address__ - regex: (.+) replacement: /api/v1/nodes/$$1/proxy/metrics source_labels: - __meta_kubernetes_node_name target_label: __metrics_path__ scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true - job_name: kubernetes-resource bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: node relabel_configs: - action: labelmap regex: __meta_kubernetes_node_label_(.+) - replacement: kubernetes.default.svc:443 target_label: __address__ - regex: (.+) replacement: /api/v1/nodes/$$1/proxy/metrics/resource source_labels: - __meta_kubernetes_node_name target_label: __metrics_path__ scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true - job_name: 'kubelet-probes' kubernetes_sd_configs: - role: node scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - action: labelmap regex: __meta_kubernetes_node_label_(.+) - target_label: __address__ replacement: kubernetes.default.svc:443 - source_labels: [__meta_kubernetes_node_name] regex: (.+) target_label: __metrics_path__ replacement: /api/v1/nodes/$$1/proxy/metrics/probes - job_name: kubernetes-nodes-cadvisor bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: node relabel_configs: - action: labelmap regex: __meta_kubernetes_node_label_(.+) - replacement: kubernetes.default.svc:443 target_label: __address__ - regex: (.+) replacement: /api/v1/nodes/$$1/proxy/metrics/cadvisor source_labels: - __meta_kubernetes_node_name target_label: __metrics_path__ scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true - job_name: kubernetes-service-endpoints kubernetes_sd_configs: - role: endpoints relabel_configs: - action: keep regex: true source_labels: - __meta_kubernetes_service_annotation_prometheus_io_scrape - action: replace regex: (https?) source_labels: - __meta_kubernetes_service_annotation_prometheus_io_scheme target_label: __scheme__ - action: replace regex: (.+) source_labels: - __meta_kubernetes_service_annotation_prometheus_io_path target_label: __metrics_path__ - action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $$1:$$2 source_labels: - __address__ - __meta_kubernetes_service_annotation_prometheus_io_port target_label: __address__ - action: labelmap regex: __meta_kubernetes_service_annotation_prometheus_io_param_(.+) replacement: __param_$$1 - action: labelmap regex: __meta_kubernetes_service_label_(.+) - action: replace source_labels: - __meta_kubernetes_namespace target_label: kubernetes_namespace - action: replace source_labels: - __meta_kubernetes_service_name target_label: kubernetes_name - action: replace source_labels: - __meta_kubernetes_pod_node_name target_label: kubernetes_node - job_name: kube-dns honor_labels: true kubernetes_sd_configs: - role: pod relabel_configs: - action: keep source_labels: - __meta_kubernetes_namespace - __meta_kubernetes_pod_name separator: '/' regex: 'kube-system/coredns.+' - source_labels: - __meta_kubernetes_pod_container_port_name action: keep regex: metrics - source_labels: - __meta_kubernetes_pod_name action: replace target_label: instance - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - job_name: prometheus-pushgateway kubernetes_sd_configs: - role: service relabel_configs: - action: keep regex: pushgateway source_labels: - __meta_kubernetes_service_annotation_prometheus_io_probe - job_name: kubernetes-pods kubernetes_sd_configs: - role: pod relabel_configs: - action: keep regex: true source_labels: - __meta_kubernetes_pod_annotation_prometheus_io_scrape - action: replace regex: (https?) source_labels: - __meta_kubernetes_pod_annotation_prometheus_io_scheme target_label: __scheme__ - action: replace regex: (.+) source_labels: - __meta_kubernetes_pod_annotation_prometheus_io_path target_label: __metrics_path__ - action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $$1:$$2 source_labels: - __address__ - __meta_kubernetes_pod_annotation_prometheus_io_port target_label: __address__ - action: labelmap regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+) replacement: __param_$$1 - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - action: replace source_labels: - __meta_kubernetes_namespace target_label: kubernetes_namespace - action: replace source_labels: - __meta_kubernetes_pod_name target_label: kubernetes_pod_name - action: drop regex: Pending|Succeeded|Failed|Completed source_labels: - __meta_kubernetes_pod_phase - job_name: kubecost honor_labels: true honor_timestamps: true scrape_interval: 30s scrape_timeout: 10s metrics_path: /metrics scheme: http follow_redirects: true dns_sd_configs: - names: - kubecost-cost-analyzer.cost-service.svc.cluster.local refresh_interval: 30s type: A port: 9003 # These entries will be removed in the future if a better alternative is found === - job_name: 'node' scrape_interval: 30s static_configs: - targets: - 'prometheus-kube-prometheus-prometheus.prometheus.svc.cluster.local:9090' - job_name: 'alert-manager' scrape_interval: 10m static_configs: - targets: - 'prometheus-kube-prometheus-alertmanager.prometheus.svc.cluster.local:9093' - job_name: 'tekton' scrape_interval: 10m static_configs: - targets: - 'tekton-pipelines-controller.tekton-pipelines.svc.cluster.local:9090' - 'tekton-pipelines-webhook.tekton-pipelines.svc.cluster.local:9090' - job_name: 'minio' bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token metrics_path: /minio/v2/metrics/cluster scheme: http static_configs: - targets: - 'minio.hitachi-solutions.svc.cluster.local:9000' - job_name: 'otel' scrape_interval: 30s static_configs: - targets: - ${env:POD_IP}:8888

Photo of Prabhat
Prabhat
Tue, 17 Oct 2023 10:06:15 UTC

Cool. Official collector does the same thing. It uses otel-collector operator to simplify things. check this config. Would love to get your feedback on this.

Photo of Jennings
Jennings
Tue, 17 Oct 2023 17:16:12 UTC

Prabhat I will take a look. West thanks for sharing your YAMLs

OpenObserve Logo

OpenObserve

Indexed 419 threads

OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI

View in App

Similar Threads

Exploring OpenObserve Dashboard and Alert Installation, Metrics, Help with Data
OTel Metrics Setup - 404 Error and Authentication Issues
Troubleshooting Installation of OpenObserve
Adding OpenObserve as Export Destination for Logs
Troubleshooting Metrics Ingestion with Prometheus RemoteWrite