topic Re: How to setup a Data Alert for missing data? in Dashboards & Analytics Discussions

How to setup a Data Alert for missing data?

mike_mascitti — Fri, 31 Mar 2023 14:44:49 GMT

Hello, I'm trying to setup a data alert when the data read from our source is not new. What happens is, sometimes our external source table has not been updated when our Incorta Schema load runs and we end up pulling in no data for 'today'. When this happens I'd like to send a data alert.

The issue is, data alerts only trigger when some data is present. I can't filter the dataset on 'data that isn't from today' because all the previous data gets returned. I can't filter on the max(effective_date) because 'max' is not supported in filter formulas..

How do I return data to trigger the alert when what I'm checking for is missing data?

Re: How to setup a Data Alert for missing data?

RADSr — Fri, 31 Mar 2023 15:28:25 GMT

Do you have a calendar table in your model? Using that with "show empty groups" might get you there.

Re: How to setup a Data Alert for missing data?

dylanwan — Fri, 31 Mar 2023 15:45:16 GMT

One possibility is to get the row count from source tables and compare the row count with the tables loaded into Incorta. We can then send the data alert when there are significant difference found.

It may hard to be exactly the same when the data can be added to the source table anytime. However, the number is the last incorta scheduled refresh should be greater than the last runcount SQL run and should be less than the next runcount SQL run. We had implemented this in a couple of incorta customers.

Re: How to setup a Data Alert for missing data?

dylanwan — Fri, 31 Mar 2023 15:49:08 GMT

The other issue you mentioned about the aggregate filter is not available in Data Alert may be solvable via the result set feature. If the aggregated result set is used, the measure showing the aggregate will be a regular column in the outside and can be placed in filter.

Re: How to setup a Data Alert for missing data?

sbeard — Tue, 26 Nov 2024 21:54:32 GMT

I have a similar situation to this. We expect exactly 83,624 records extracted everyday at the same time for a certain load. Sometimes, this data load will load successfully, but extract nothing, and I hope to find a way to get a notification sent to me when a load of this table extracts 0 records. It sounds like this could be done with this method you are describing as we expect a specific amount and the situation I want to get notified for is also a specific, much smaller amount. You say you have implemented this with a couple customers and I would also like to implement this.