Integrating ZO Monitoring with PromQL and Vector
TLDR Gaby suggests bundling ZO monitoring features by default. Prabhat explains that PromQL will be supported in an upcoming release, and provides instructions for integrating with Vector. They discuss improvements and potential documentation updates.
Prabhat
Thu, 25 May 2023 13:42:10 UTCPromQL support will be out in the coming release. Bundling charts or picking from a git repo or any other repo should be follow on items.
Prabhat
Thu, 25 May 2023 13:42:30 UTCThese charts are built using PromQL
Gaby
Thu, 25 May 2023 13:43:10 UTCInteresting, i have to read about PromQL
Gaby
Thu, 25 May 2023 13:43:49 UTCDoes that mean it requires having a separate grafana/prometheus instance? Or is it built into ZO?
Prabhat
Thu, 25 May 2023 13:45:17 UTCSo ZO exposes its metrics using prometheus endpoints. Try visiting
Prabhat
Thu, 25 May 2023 13:45:27 UTCYou don't need grafana
Prabhat
Thu, 25 May 2023 13:46:07 UTCYou do however need prometheus or otel collector to scrape these metrics and feed them back in ZincObserve for now.
Gaby
Thu, 25 May 2023 13:47:24 UTCOhhh, might be worth adding in the docs how to do that with Vector, since I'm pretty sure it can do that easily. Users can then just copy paste the toml into their vector instance
Gaby
Thu, 25 May 2023 13:47:49 UTC:joy: if i figured something out, i will share it
Prabhat
Thu, 25 May 2023 13:48:26 UTCActually vector can scrape prometheus metrics
Prabhat
Thu, 25 May 2023 13:49:00 UTCDo you have a multi node ZincObserve setup?
Prabhat
Thu, 25 May 2023 13:49:50 UTCif its a single node setup then then its as simple as
```[sources.my_source_id]
type = "prometheus_scrape"
endpoints = [ "
Prabhat
Thu, 25 May 2023 13:50:05 UTCfor source
Prabhat
Thu, 25 May 2023 13:50:09 UTCand for target
Gaby
Thu, 25 May 2023 13:50:34 UTCYeah i only have 1 ZO instance, and 1 Vector
Gaby
Thu, 25 May 2023 13:51:02 UTCAwesome, i will give that a try once the new version is out :-)
Prabhat
Thu, 25 May 2023 13:51:14 UTC```[sinks.my_sink_id]
type = "prometheus_remote_write"
inputs = [ "my_source_id" ]
endpoint = "
Prabhat
Thu, 25 May 2023 13:51:23 UTCthats it and you are done
Prabhat
Thu, 25 May 2023 13:52:08 UTCAre you running it in kubernetes?
Gaby
Thu, 25 May 2023 13:52:12 UTCPlain Docker
Gaby
Thu, 25 May 2023 13:52:21 UTCWell Docker Compose :joy:
Prabhat
Thu, 25 May 2023 13:52:37 UTCwhy docker compose. Its a single container
Gaby
Thu, 25 May 2023 13:53:20 UTCBoth ZO and Vector are on the same host
Gaby
Thu, 25 May 2023 13:53:40 UTCAlso NGINX for https
Prabhat
Thu, 25 May 2023 13:53:57 UTCoh, you have your homelab fully setup
Gaby
Thu, 25 May 2023 13:54:18 UTCBut vector can publish to ZO via http, within the docker network without the https overhead
Gaby
Thu, 25 May 2023 13:54:24 UTCYeah, this is a home lab setup
Gaby
Thu, 25 May 2023 13:54:50 UTCAt work we only testing ZO, the prod metrics are in ELK still
Prabhat
Thu, 25 May 2023 13:55:38 UTCHow many nodes of ES do you have at work?
Gaby
Thu, 25 May 2023 13:56:49 UTCI think 6-8, one vm for kibana, one vm for logstash
Gaby
Thu, 25 May 2023 13:57:07 UTCWe are mirroring data to ZO. And only 1 VM of ZO can handle the same amount :joy:
Gaby
Thu, 25 May 2023 13:57:29 UTCAround 2-4million logs/hr
Prabhat
Thu, 25 May 2023 13:58:34 UTCthat is crazy. We have more performance improvements coming soon.
Gaby
Thu, 25 May 2023 13:58:52 UTCYeah the low mem usage, is insane
Gaby
Thu, 25 May 2023 13:59:01 UTCELK eats ram like its candy
Prabhat
Thu, 25 May 2023 13:59:28 UTCYeah, JVM is like a monster.
Gaby
Wed, 07 Jun 2023 14:21:05 UTCPrabhat Would be useful to have the PromQL stuff with Vector as part of the docs :-)
Prabhat
Wed, 07 Jun 2023 14:28:59 UTCYeah, that will be good.
Prabhat
Wed, 07 Jun 2023 14:30:35 UTCWe have been thinking of a different approach though. Instead of piecemeal approach around tools, we provide a single solution that works for multiple scenarios in a particular environment. e.g. k8s. You get a setup that does everything, ec2 a setup, ECS another one, GCP another one, VMWare another .
Gaby
Wed, 07 Jun 2023 14:33:38 UTCHmmm interesting, not sure how that would work for folks with custom setups or just running Docker/Syslog/Custom Services
Gaby
Wed, 07 Jun 2023 14:36:46 UTCThere's also the case where you are running in a mixed environment between baremetal + cloud
Prabhat
Wed, 07 Jun 2023 14:58:32 UTCYeah, We will have to figure these things out. We are biting faster than we could chew.
OpenObserve
Indexed 419 threads
OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI
View in App
Gaby
Thu, 25 May 2023 13:39:47 UTCIt would be really useful if ZO came bundled with this dashboard from this issue by default under a "Monitoring" section.