Mäkelä, Antti
2018-10-15 07:50:20 UTC
Hi,
What would be the best method to have SMS notifications sent only during business hours, *including* if service/host is down at the start?
The problem:
- Business hours are 7 AM to 10 PM.
- Host notification delay is 5 minutes, so that momentary "flaps" are not causing notifications.
- Host goes down at 3 AM and comes back up at 4 AM. No notifications should be generated.
- Host goes down at 6 AM and is still down at 7 AM. Notifications should be generated at 7 AM (not at 7:05).
- Host goes down at 8 AM. Notification should be generated at 8:05 AM.
There are at least three different parameters here:
- Check period for hosts.
- Notification period for hosts
- Service period for hosts
- Recurring downtimes.
We've not tested all possible situations yet, but it seems that when testing the scenarion where clock reaches 7 AM and host goes down during the night (and stays down).
- Adjusting Check period for hosts to 7AM - 10 PM => Causes the alert to come in at 7:05 since checks are not even initiated until 7 AM. (Workaround: Adjust from 10 PM to 6:55 AM). Furthermore, availability reports might be skewed if the host goes down at 21:55 and comes up right after 10 PM => the entire night is marked as "down".
- Adjusting notification period for hosts to 7AM - 10 PM => Alerts are not coming in at all, since the actual DOWN transition happened outside of notification period.
- Service period for hosts => Only affects availability reporting it seems.
- Recurring downtime => Not tested yet. Furthermore, possibly affects availability reports.
Anyway, what would be the best way to achieven the desired result? We don't want to wake our on-call people for things that don't essentially matter until morning in case things recover on their own. However, once business hours are started, the notifications should be sent immediately.
What would be the best method to have SMS notifications sent only during business hours, *including* if service/host is down at the start?
The problem:
- Business hours are 7 AM to 10 PM.
- Host notification delay is 5 minutes, so that momentary "flaps" are not causing notifications.
- Host goes down at 3 AM and comes back up at 4 AM. No notifications should be generated.
- Host goes down at 6 AM and is still down at 7 AM. Notifications should be generated at 7 AM (not at 7:05).
- Host goes down at 8 AM. Notification should be generated at 8:05 AM.
There are at least three different parameters here:
- Check period for hosts.
- Notification period for hosts
- Service period for hosts
- Recurring downtimes.
We've not tested all possible situations yet, but it seems that when testing the scenarion where clock reaches 7 AM and host goes down during the night (and stays down).
- Adjusting Check period for hosts to 7AM - 10 PM => Causes the alert to come in at 7:05 since checks are not even initiated until 7 AM. (Workaround: Adjust from 10 PM to 6:55 AM). Furthermore, availability reports might be skewed if the host goes down at 21:55 and comes up right after 10 PM => the entire night is marked as "down".
- Adjusting notification period for hosts to 7AM - 10 PM => Alerts are not coming in at all, since the actual DOWN transition happened outside of notification period.
- Service period for hosts => Only affects availability reporting it seems.
- Recurring downtime => Not tested yet. Furthermore, possibly affects availability reports.
Anyway, what would be the best way to achieven the desired result? We don't want to wake our on-call people for things that don't essentially matter until morning in case things recover on their own. However, once business hours are started, the notifications should be sent immediately.
--
- Dr. Antti Mäkelä | Senior Architect | CCIE #20962 -
- Vintor Oy, Itsehallintokuja 6, 02600 Espoo | www.vintor.fi -
- Dr. Antti Mäkelä | Senior Architect | CCIE #20962 -
- Vintor Oy, Itsehallintokuja 6, 02600 Espoo | www.vintor.fi -