#openobserve

Discussion on Real-Time Alerts and Conditions

TLDR Jurgen wanted alerts for instance value changes in metrics. Prabhat recommended using external tools as OpenObserve currently lacks this functionality. Ashish confirmed the lack of complete data in alerts and promised potential enhancement in OpenObserve.

5mo

Join the chat

Jul 05, 2023 (5 months ago)

Photo of md5-0c89553a689ed850edefe8ffeae8ab1a

Jurgen

09:08 AM

Question about real-time alerts: I'd like to trigger an alert when the value went from 1 to 0 on the metrics probe_success on any specific instance. Currently, I only seem to be able to raise an alert when value=0, but that raises the same alert every few seconds when blackbox pings the service every few seconds. Is that possible somehow?

Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5

Prabhat

09:29 AM

We don't have support for multiple conditions for now. I have added an issue to implement it - https://github.com/openobserve/openobserve/issues/1101

Jurgen

09:50 AM

I'm not sure, if multiple conditions will resolve the issue. What we need instead is when a value changes for an instance. So we can not only query records with multiple conditions, we would also see if the value got changed when compared to the previous record for the same instance. Do you think, multiple conditions will allow to do that?

Prabhat

01:22 PM

So these are 2 different things:
1. Alerting based on a condition happening
2. Alerting based on change condition happening and then reverting. eg. status was up (1) and then changed to down (0) we want to get alerted. We also want to get alerted when the system came back up.

01:28

Prabhat

01:28 PM

Requirement 2 is only scratching the surface in terms of sophisticated alerting. There are features that one might need like silencing alerts , grouping them, inhibiting them. That is a lot of work and OpenObserve is unlikely to get these features in short term. There are specific tools built specifically to manage alerts and they go very deep in them. Prometheus alertmanager (https://prometheus.io/docs/alerting/latest/alertmanager/) is such a tool. What we recommend is to setup Prometheus alertmanager which will receive alerts from OpenObserve and you can do a lot of these advanced things there. You could also use keep (https://github.com/keephq/keep - I haven't tested it though).

Jurgen

02:08 PM

OK, I'll have a look. Although, I would have thought that alerting on changed values only would be a basic requirement. Will see, how that's being addressed in AlertManager.

Prabhat

02:08 PM

I noticed that this is a standard feature in kibana

02:08

Prabhat

02:08 PM

where you are coming from

Jurgen

02:09 PM

We currently use Alerta (https://alerta.io/) as our alert dashboard, which is really great. But we need changed value alerts only.

Prabhat

02:10 PM

Did not know about alerta. This looks good.

02:13

Prabhat

02:13 PM

In prometeus alertmanager you would want to use

send_resolved: true

for enabling this

Jul 06, 2023 (5 months ago)

Jurgen

07:36 AM

Started playing with Alert Manager, at least it's really simple for initial setup. But problem seems to be that the content of an alert being sent by OpenObserve doesn't contain any data which would allow us to identify what's wrong. The way I read it, these are the only variables for templates:
• stream_name
• org_name
• alert_name
• alert_type
• timestamp
Is that correct? Or is there a way to access data from the row which triggered the alert?

Photo of md5-9ed257a93c49bf4a991f872cc2ea4cda

Ashish

10:36 AM

Jurgen yes..as of today we send this data only

10:37

Ashish

10:37 AM

For real time alerts we can send data which has resulted in alert

10:37

Ashish

10:37 AM

for scheduled alert as it is over time window

10:38

Ashish

10:38 AM

would start and end time of duration help?

Jurgen

10:39 AM

I'm currently working with real time alerts. Are you saying I can then already send more data?

Ashish

10:40 AM

not as of now…it would need enhancement in OpenObserve

10:40

Ashish

10:40 AM

one more question

10:41

Ashish

10:41 AM

today if you are ingesting 10 records in single request..from these 10 if 6 are meeting alert codition

10:41

Ashish

10:41 AM

would you want to get all 6 of them or just last one from request

Jurgen

11:05 AM

I guess we want them all. In our use case we receive data from Prometheus Blackbox and if any of the records has probe_success=0 we want an alert being issued. I assume that Prometheus sends records from a number of different probes at once, so each of which would have to raise an alert, if they were not successful

Ashish

11:18 AM

sure..we will discuss & come back to you

OpenObserve

OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI

Indexed 406 threads (74% resolved)

Join Our Community

Similar Threads

Erroneous Triggering of Alarm in 0.5.1

Chris expressed experiencing occasional erroneous alerts trigger on a specific stream query. Ashish suggested the issue might be linked to a known duplicates bug. Uncertain, Chris decided to monitor the issue further and report if it persists.

3mo

ZincObserve Installation Issue - Email and Env variables

Jens faced issues after installing ZincObserve via Helm chart due to `ZO_ROOT_USER_EMAIL` and `ZO_ROOT_USER_PASSWORD` not set. Prabhat suggested trying the latest dev container and changing to a simpler password. The issue was resolved after changing the email which didn't match the regex validation.

123

6mo

ZincSearch vs Kibana and OpenObserve Improvements

Gaby is looking to move away from ElasticSearch, but finds OpenObserve lacks features he needs. Prabhat and Kirtan discuss potential improvements and consider user feedback valuable.

Understanding and Integrating with OpenObserve

刘希晨 sought clarification on the difference between ZincSearch, ZincObserve, and OpenObserve. Prabhat responded and actively engaged in a long discussion regarding an integration for 刘希晨's microservice framework. Hengfei shared that the issue 刘希晨 faced with traces upload was acknowledged and fixed.

5mo

Issue with Alerts on Metric "testing" not Triggering

Mark encountered an issue with alert creation on "testing" metric, and Ashish identified a bug, which was fixed in the released version v0.4.2.

7mo