I've got this table - it's huge, not by data volume standards ( ~ 45 million records and growing w/ projected exponential growth ), but by load-time standards ( ~ 25 hours to do a full load ).
I need a way to speed this up. ( I'm already doing incremental loads but situations arise where a full load is needed )
*** I have no influence over the source DB whatsoever, so "tune the view," while good advice, is not something I can do -- non starter **
Anyway - the ~ 25 hour load time is when I select * from view.
I did a test breaking that into 12 tables and adding a where clause to each to bring in a year's worth of data in ~ 2 hours. Huzzah!
So I can create a derived table to UNION those, right? <BZZZZZTTTTT> Incorta SQL doesn't like the DECIMAL data type.
Decimal type is expected for select item: Txxxxxxxx
Currently decimal type is not supported!
Not supported query
OK - so the very quick engine isn't available, but surely a SQLi table will work ...
- INC_03010100: Unknown Error: Extraction failed [com.incorta.connector.errors.DiscoveryException: Error of type PSQLException while discovering columns from datasource [SQLInterface] due to: ERROR: Decimal type is expected for select item:
So I'm learning that combining two Incorta tables, created by Incorta, with the Incorta assigned data types isn't as straightforward as one might expect.
I'm out for the evening, but if anyone has any experience w/ this type of exercise and can point me in the right direction I'm happy to learn!