TLDR Gaby suggests bundling ZO monitoring features by default. Prabhat explains that PromQL will be supported in an upcoming release, and provides instructions for integrating with Vector. They discuss improvements and potential documentation updates.
PromQL support will be out in the coming release. Bundling charts or picking from a git repo or any other repo should be follow on items.
These charts are built using PromQL
Interesting, i have to read about PromQL
Does that mean it requires having a separate grafana/prometheus instance? Or is it built into ZO?
So ZO exposes its metrics using prometheus endpoints. Try visiting
You don't need grafana
You do however need prometheus or otel collector to scrape these metrics and feed them back in ZincObserve for now.
Ohhh, might be worth adding in the docs how to do that with Vector, since I'm pretty sure it can do that easily. Users can then just copy paste the toml into their vector instance
:joy: if i figured something out, i will share it
Actually vector can scrape prometheus metrics
Do you have a multi node ZincObserve setup?
if its a single node setup then then its as simple as
```[sources.my_source_id]
type = "prometheus_scrape"
endpoints = [ "
for source
and for target
Yeah i only have 1 ZO instance, and 1 Vector
Awesome, i will give that a try once the new version is out :-)
```[sinks.my_sink_id]
type = "prometheus_remote_write"
inputs = [ "my_source_id" ]
endpoint = "
thats it and you are done
Are you running it in kubernetes?
Plain Docker
Well Docker Compose :joy:
why docker compose. Its a single container
Both ZO and Vector are on the same host
Also NGINX for https
oh, you have your homelab fully setup
But vector can publish to ZO via http, within the docker network without the https overhead
Yeah, this is a home lab setup
At work we only testing ZO, the prod metrics are in ELK still
How many nodes of ES do you have at work?
I think 6-8, one vm for kibana, one vm for logstash
We are mirroring data to ZO. And only 1 VM of ZO can handle the same amount :joy:
Around 2-4million logs/hr
that is crazy. We have more performance improvements coming soon.
Yeah the low mem usage, is insane
ELK eats ram like its candy
Yeah, JVM is like a monster.
Prabhat Would be useful to have the PromQL stuff with Vector as part of the docs :-)
Yeah, that will be good.
We have been thinking of a different approach though. Instead of piecemeal approach around tools, we provide a single solution that works for multiple scenarios in a particular environment. e.g. k8s. You get a setup that does everything, ec2 a setup, ECS another one, GCP another one, VMWare another .
Hmmm interesting, not sure how that would work for folks with custom setups or just running Docker/Syslog/Custom Services
There's also the case where you are running in a mixed environment between baremetal + cloud
Yeah, We will have to figure these things out. We are biting faster than we could chew.
Gaby
Thu, 25 May 2023 13:39:47 UTCIt would be really useful if ZO came bundled with this dashboard from this issue by default under a "Monitoring" section.