0
-
Are you talking about extracting data from source database with UNION?
I do not run into SQL union issue. However, if we want to make it run faster, one option we can consider is to put the logic as multiple data sets against the same source database.Please note that when we use multiple data sets, each will be considered as a separate extraction and will be run in parallel. It also means that the connection pool against the data source should be set to a higher number if we are creating a large number of data sets.
The SQL extraction query performance can be further divided into multiple chunks by using the chunking feature if you are using Oracle or MySQL. We are working on the feature to make it available to other database.
Hope this helps.
-
Hi Dylan,
thank you for your answer.
When I directly ran the SQL in Incorta , I have no issue.
I have been told by my Incorta adminnistrator, that I should first :
Create a Table with all my data inside, for example :
Create a incorta TABLE "TEST" with SELECT * from mytable
-> is that the best practice?
Then, create another table with SELECT Col1, Col2 from TEST where Col3 is not null UNION SELECT Col1, Col2 from TEST where Col4 > 4 etc....etc... but then it 's running forever and I end up with a timeout issue.
If I directly use "SELECT Col1, Col2 from mytable where Col3 is not null UNION SELECT Col1, Col2 from mytable where Col4 > 4" it works fast.
-
Luigi Riccio Are you creating materialized views or a SQL database in your second query?
It should work if you are using Materialized View.
-
Dylan Wan Hi there.
surprisingly it is working with Materialized View. :). I haven't really figured out what's the difference between Materialized View or an SQL statement querying the incorta tables but I am happy it works , thank you
-
-
Doing union in Materialized View or sql union query using sqli connection (using postgres adaptor to query incorta on port 5436) should work and both will use spark.
