#openobserve

Discussion on Real-Time Alerts and Conditions

TLDR Jurgen wanted alerts for instance value changes in metrics. Prabhat recommended using external tools as OpenObserve currently lacks this functionality. Ashish confirmed the lack of complete data in alerts and promised potential enhancement in OpenObserve.

Powered by Struct AI

1

Jul 05, 2023 (5 months ago)
Jurgen
Photo of md5-0c89553a689ed850edefe8ffeae8ab1a
Jurgen
09:08 AM
Question about real-time alerts: I'd like to trigger an alert when the value went from 1 to 0 on the metrics probe_success on any specific instance. Currently, I only seem to be able to raise an alert when value=0, but that raises the same alert every few seconds when blackbox pings the service every few seconds. Is that possible somehow?
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
09:29 AM
We don't have support for multiple conditions for now. I have added an issue to implement it - https://github.com/openobserve/openobserve/issues/1101
Jurgen
Photo of md5-0c89553a689ed850edefe8ffeae8ab1a
Jurgen
09:50 AM
I'm not sure, if multiple conditions will resolve the issue. What we need instead is when a value changes for an instance. So we can not only query records with multiple conditions, we would also see if the value got changed when compared to the previous record for the same instance. Do you think, multiple conditions will allow to do that?
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
01:22 PM
So these are 2 different things:
1. Alerting based on a condition happening
2. Alerting based on change condition happening and then reverting. eg. status was up (1) and then changed to down (0) we want to get alerted. We also want to get alerted when the system came back up.
01:28
Prabhat
01:28 PM
Requirement 2 is only scratching the surface in terms of sophisticated alerting. There are features that one might need like silencing alerts , grouping them, inhibiting them. That is a lot of work and OpenObserve is unlikely to get these features in short term. There are specific tools built specifically to manage alerts and they go very deep in them. Prometheus alertmanager (https://prometheus.io/docs/alerting/latest/alertmanager/) is such a tool. What we recommend is to setup Prometheus alertmanager which will receive alerts from OpenObserve and you can do a lot of these advanced things there. You could also use keep (https://github.com/keephq/keep - I haven't tested it though).
Jurgen
Photo of md5-0c89553a689ed850edefe8ffeae8ab1a
Jurgen
02:08 PM
OK, I'll have a look. Although, I would have thought that alerting on changed values only would be a basic requirement. Will see, how that's being addressed in AlertManager.
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
02:08 PM
I noticed that this is a standard feature in kibana
02:08
Prabhat
02:08 PM
where you are coming from
Jurgen
Photo of md5-0c89553a689ed850edefe8ffeae8ab1a
Jurgen
02:09 PM
We currently use Alerta (https://alerta.io/) as our alert dashboard, which is really great. But we need changed value alerts only.
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
02:10 PM
Did not know about alerta. This looks good.
02:13
Prabhat
02:13 PM
In prometeus alertmanager you would want to use
send_resolved: true

for enabling this
Jul 06, 2023 (5 months ago)
Jurgen
Photo of md5-0c89553a689ed850edefe8ffeae8ab1a
Jurgen
07:36 AM
Started playing with Alert Manager, at least it's really simple for initial setup. But problem seems to be that the content of an alert being sent by OpenObserve doesn't contain any data which would allow us to identify what's wrong. The way I read it, these are the only variables for templates:
• stream_name
• org_name
• alert_name
• alert_type
• timestamp
Is that correct? Or is there a way to access data from the row which triggered the alert?
Ashish
Photo of md5-9ed257a93c49bf4a991f872cc2ea4cda
Ashish
10:36 AM
Jurgen yes..as of today we send this data only
10:37
Ashish
10:37 AM
For real time alerts we can send data which has resulted in alert
10:37
Ashish
10:37 AM
for scheduled alert as it is over time window
10:38
Ashish
10:38 AM
would start and end time of duration help?
Jurgen
Photo of md5-0c89553a689ed850edefe8ffeae8ab1a
Jurgen
10:39 AM
I'm currently working with real time alerts. Are you saying I can then already send more data?
Ashish
Photo of md5-9ed257a93c49bf4a991f872cc2ea4cda
Ashish
10:40 AM
not as of now…it would need enhancement in OpenObserve
10:40
Ashish
10:40 AM
one more question
10:41
Ashish
10:41 AM
today if you are ingesting 10 records in single request..from these 10 if 6 are meeting alert codition
10:41
Ashish
10:41 AM
would you want to get all 6 of them or just last one from request
Jurgen
Photo of md5-0c89553a689ed850edefe8ffeae8ab1a
Jurgen
11:05 AM
I guess we want them all. In our use case we receive data from Prometheus Blackbox and if any of the records has probe_success=0 we want an alert being issued. I assume that Prometheus sends records from a number of different probes at once, so each of which would have to raise an alert, if they were not successful
Ashish
Photo of md5-9ed257a93c49bf4a991f872cc2ea4cda
Ashish
11:18 AM
sure..we will discuss & come back to you

1

OpenObserve

OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI

Indexed 406 threads (74% resolved)

Join Our Community