#openobserve

Understanding Alerts and Webhook Destinations in OpenObserve API

TLDR Mik inquired about alerts and their triggers and destination in the OpenObserve API. Ashish confirmed that other destinations are supported, showed how to verify when alerts are triggered, and discussed the variables currently available in the alert's template.

2mo

Join the chat

Oct 05, 2023 (2 months ago)

Photo of md5-831a5d6be712944970cc91a59491f238

Mik

01:43 AM

Hi, everyone. Just a few questions:
• How do I know if the alert that I've created is being triggered?
• ~~The documentation says that "Please note we only support slack webhook as destination for both kind of alerts." - I presume that I cannot create a custom endpoint for destination?~~ (I've tested this in OpenObserve API, working for other destination)

Photo of md5-9ed257a93c49bf4a991f872cc2ea4cda

Ashish

03:24 AM

Hi Mik

03:24

Ashish

03:24 AM

Yes we support any destination with webhook...our docs need update

03:25

Ashish

03:25 AM

To check if alert is triggered .there are two ways..in case condition is fulfilled you would receive a notification

03:26

Ashish

03:26 AM

What kind of deployment you have Single node or HA ...which type of alert did you create real time or scheduled?

03:26

Ashish

03:26 AM

For scheduled alert you can check alert manager logs

Mik

06:39 AM

Thanks Ashish currently we have a single node setup. The problem is it seems that the condition that I've made for the alert isn't being triggered :thinking_face:

Could you please point me to the Alert manager logs, I can't locate it

Ashish

07:34 AM

since your is single node setup

07:34

Ashish

07:34 AM

the logs would be part of openobsrve log

07:34

Ashish

07:34 AM

in single node setup..same node plays all roles

Mik

08:23 AM

Thanks a lot Ashish follow up question, is there a way that we can include other variable in the alert's template? something like the query and/or the condition of the alert

Ashish

08:24 AM

as of today nothing except : stream_name
org_name
alert_name
alert_type
timestamp

08:24

Ashish

08:24 AM

we have in plan to include more…

Mik

08:25 AM

Thanks heaps!

OpenObserve

OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI

Indexed 404 threads (74% resolved)

Join Our Community

Similar Threads

Discussion on Real-Time Alerts and Conditions

Jurgen wanted alerts for instance value changes in metrics. Prabhat recommended using external tools as OpenObserve currently lacks this functionality. Ashish confirmed the lack of complete data in alerts and promised potential enhancement in OpenObserve.

5mo

Erroneous Triggering of Alarm in 0.5.1

Chris expressed experiencing occasional erroneous alerts trigger on a specific stream query. Ashish suggested the issue might be linked to a known duplicates bug. Uncertain, Chris decided to monitor the issue further and report if it persists.

3mo

Issue with Alerts on Metric "testing" not Triggering

Mark encountered an issue with alert creation on "testing" metric, and Ashish identified a bug, which was fixed in the released version v0.4.2.

7mo

Understanding and Integrating with OpenObserve

刘希晨 sought clarification on the difference between ZincSearch, ZincObserve, and OpenObserve. Prabhat responded and actively engaged in a long discussion regarding an integration for 刘希晨's microservice framework. Hengfei shared that the issue 刘希晨 faced with traces upload was acknowledged and fixed.

5mo

Vector Crashes with Timeouts during ZO Ingestion

Gaby experienced Vector crashes and timeouts during ZO ingestion. Prabhat suggested disabling alerts, and Gaby found better performance without alerts and by adjusting batch and buffer settings.

6mo