We have a process where we have an internal group drop files multiple times daily to our GlobalScape (GS) system and those files get loaded daily schedule to an Incorta table we have set up, via a data lake data source. The problem is that the files don't have an exact schedule. They will stop dropping to GS around the 4th of the month, and pick up again around the 17th. Our GS queue dumps files after 7 days, so at a point in that stretch, we end up with no files in the queue, and we start getting load errors in Incorta. From what I've been able to tell from looking into it, there is no option for data lakes to add Extra Options values "ignoreMissingFiles=true" or "failOnEmpty=false (to avoid errors when no files are found)". My enhancement request would be to make those options available for data lake connectors. We want to be able to ignore if there are no files and not generate any errors. (Having an extra file dropped in the empty data file period is possible, but not ideal.)
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.