Integrating ZO Monitoring with PromQL and Vector

TLDR Gaby suggests bundling ZO monitoring features by default. Prabhat explains that PromQL will be supported in an upcoming release, and provides instructions for integrating with Vector. They discuss improvements and potential documentation updates.

Photo of Gaby
Gaby
Thu, 25 May 2023 13:39:47 UTC

It would be really useful if ZO came bundled with this dashboard from this issue by default under a "Monitoring" section.

Photo of Prabhat
Prabhat
Thu, 25 May 2023 13:42:10 UTC

PromQL support will be out in the coming release. Bundling charts or picking from a git repo or any other repo should be follow on items.

Photo of Prabhat
Prabhat
Thu, 25 May 2023 13:42:30 UTC

These charts are built using PromQL

Photo of Gaby
Gaby
Thu, 25 May 2023 13:43:10 UTC

Interesting, i have to read about PromQL

Photo of Gaby
Gaby
Thu, 25 May 2023 13:43:49 UTC

Does that mean it requires having a separate grafana/prometheus instance? Or is it built into ZO?

Photo of Prabhat
Prabhat
Thu, 25 May 2023 13:45:17 UTC

So ZO exposes its metrics using prometheus endpoints. Try visiting on your laptop or change the domain to appropriate installation and you will see all the metrics of ZO in prometheus format.

Photo of Prabhat
Prabhat
Thu, 25 May 2023 13:45:27 UTC

You don't need grafana

Photo of Prabhat
Prabhat
Thu, 25 May 2023 13:46:07 UTC

You do however need prometheus or otel collector to scrape these metrics and feed them back in ZincObserve for now.

Photo of Gaby
Gaby
Thu, 25 May 2023 13:47:24 UTC

Ohhh, might be worth adding in the docs how to do that with Vector, since I'm pretty sure it can do that easily. Users can then just copy paste the toml into their vector instance

Photo of Gaby
Gaby
Thu, 25 May 2023 13:47:49 UTC

:joy: if i figured something out, i will share it

Photo of Prabhat
Prabhat
Thu, 25 May 2023 13:48:26 UTC

Actually vector can scrape prometheus metrics

Photo of Prabhat
Prabhat
Thu, 25 May 2023 13:49:00 UTC

Do you have a multi node ZincObserve setup?

Photo of Prabhat
Prabhat
Thu, 25 May 2023 13:49:50 UTC

if its a single node setup then then its as simple as ```[sources.my_source_id] type = "prometheus_scrape" endpoints = [ "" ]```

Photo of Prabhat
Prabhat
Thu, 25 May 2023 13:50:05 UTC

for source

Photo of Prabhat
Prabhat
Thu, 25 May 2023 13:50:09 UTC

and for target

Photo of Gaby
Gaby
Thu, 25 May 2023 13:50:34 UTC

Yeah i only have 1 ZO instance, and 1 Vector

Photo of Gaby
Gaby
Thu, 25 May 2023 13:51:02 UTC

Awesome, i will give that a try once the new version is out :-)

Photo of Prabhat
Prabhat
Thu, 25 May 2023 13:51:14 UTC

```[sinks.my_sink_id] type = "prometheus_remote_write" inputs = [ "my_source_id" ] endpoint = ""```

Photo of Prabhat
Prabhat
Thu, 25 May 2023 13:51:23 UTC

thats it and you are done

Photo of Prabhat
Prabhat
Thu, 25 May 2023 13:52:08 UTC

Are you running it in kubernetes?

Photo of Gaby
Gaby
Thu, 25 May 2023 13:52:12 UTC

Plain Docker

Photo of Gaby
Gaby
Thu, 25 May 2023 13:52:21 UTC

Well Docker Compose :joy:

Photo of Prabhat
Prabhat
Thu, 25 May 2023 13:52:37 UTC

why docker compose. Its a single container

Photo of Gaby
Gaby
Thu, 25 May 2023 13:53:20 UTC

Both ZO and Vector are on the same host

Photo of Gaby
Gaby
Thu, 25 May 2023 13:53:40 UTC

Also NGINX for https

Photo of Prabhat
Prabhat
Thu, 25 May 2023 13:53:57 UTC

oh, you have your homelab fully setup

Photo of Gaby
Gaby
Thu, 25 May 2023 13:54:18 UTC

But vector can publish to ZO via http, within the docker network without the https overhead

Photo of Gaby
Gaby
Thu, 25 May 2023 13:54:24 UTC

Yeah, this is a home lab setup

Photo of Gaby
Gaby
Thu, 25 May 2023 13:54:50 UTC

At work we only testing ZO, the prod metrics are in ELK still

Photo of Prabhat
Prabhat
Thu, 25 May 2023 13:55:38 UTC

How many nodes of ES do you have at work?

Photo of Gaby
Gaby
Thu, 25 May 2023 13:56:49 UTC

I think 6-8, one vm for kibana, one vm for logstash

Photo of Gaby
Gaby
Thu, 25 May 2023 13:57:07 UTC

We are mirroring data to ZO. And only 1 VM of ZO can handle the same amount :joy:

Photo of Gaby
Gaby
Thu, 25 May 2023 13:57:29 UTC

Around 2-4million logs/hr

Photo of Prabhat
Prabhat
Thu, 25 May 2023 13:58:34 UTC

that is crazy. We have more performance improvements coming soon.

Photo of Gaby
Gaby
Thu, 25 May 2023 13:58:52 UTC

Yeah the low mem usage, is insane

Photo of Gaby
Gaby
Thu, 25 May 2023 13:59:01 UTC

ELK eats ram like its candy

Photo of Prabhat
Prabhat
Thu, 25 May 2023 13:59:28 UTC

Yeah, JVM is like a monster.

Photo of Gaby
Gaby
Wed, 07 Jun 2023 14:21:05 UTC

Prabhat Would be useful to have the PromQL stuff with Vector as part of the docs :-)

Photo of Prabhat
Prabhat
Wed, 07 Jun 2023 14:28:59 UTC

Yeah, that will be good.

Photo of Prabhat
Prabhat
Wed, 07 Jun 2023 14:30:35 UTC

We have been thinking of a different approach though. Instead of piecemeal approach around tools, we provide a single solution that works for multiple scenarios in a particular environment. e.g. k8s. You get a setup that does everything, ec2 a setup, ECS another one, GCP another one, VMWare another .

Photo of Gaby
Gaby
Wed, 07 Jun 2023 14:33:38 UTC

Hmmm interesting, not sure how that would work for folks with custom setups or just running Docker/Syslog/Custom Services

Photo of Gaby
Gaby
Wed, 07 Jun 2023 14:36:46 UTC

There's also the case where you are running in a mixed environment between baremetal + cloud

Photo of Prabhat
Prabhat
Wed, 07 Jun 2023 14:58:32 UTC

Yeah, We will have to figure these things out. We are biting faster than we could chew.