TL/DR - notify me if the loader service is down and my scheduled load didn't run.
I was recently awakened to an email asking why our data wasn't there ( *some* data was there, but not the data from the incremental build which should have happened overnight ).
I quickly checked my inbox - no failure notifications.
I then jumped into Incorta and with bleary eyes looked at the top of the schema page - no errors there ...
I looked a bit closer - the date was two days ago. That's not right.... what????
Turns out the loader service was down. Apparently the load didn't *fail* because the load never started. Incorta should loop through the schedule checking for status after the scheduled run times and include NULL status ( i.e. when the load didn't run at all ) in the "I need to email somebody" algorithm.
Bonus - maybe ping analytic, loader, and notebook services and let me know whenever any of them are down. Super-bonus - fix them before I even know they are down 😉