topic Re: Incorta SQL table cannot query from Incorta SQL ? in Administrative Discussions

Incorta SQL table cannot query from Incorta SQL ?

RADSr — Wed, 05 Jul 2023 14:54:15 GMT

We're getting an error when trying to query a table written via Incorta SQL

The actual error is that we can't query a table written in Incorta SQL in the same schema, but we get the same error when querying from a new schema so... maybe the error message isn't as accurate as it could be 😉

AFAIK an Incorta SQL table still produces a parquet file(s), no? As such, why would there be a limit on querying that table? If there is, what's the suggested workaround?

Adding: This specific case is from w/in a notebook, but we've seen the same errors elsewhere.

Re: Incorta SQL table cannot query from Incorta SQL ?

anurag — Wed, 05 Jul 2023 15:18:10 GMT

One reason it is not enabled is because memory usage by incorta SQL table can be pretty high. It loads all the columns used in SQL from the source table to memory and then loads output of the query to memory.

Re: Incorta SQL table cannot query from Incorta SQL ?

RADSr — Wed, 05 Jul 2023 15:30:01 GMT

Thank you @anurag - I am a bit confused though - If I create a derived table using Incorta SQL does it not create the parquet file? It acts similarly to an analyzer table resident only in memory?

That might go quite some way in explaining other issues I'm seeing w/ the SQLi node failing ( going into "not responding" state ).

Regardless, if I do want/need to do some complex manipulation ( build table A, query A and do some stuff - > table B, query table B and do some stuff -> C ... ) will using a SQLi data source instead of the derived table w/ Incorta SQL get me around all that RAM usage?

Re: Incorta SQL table cannot query from Incorta SQL ?

anurag — Thu, 06 Jul 2023 18:39:32 GMT

@RADSr

Derived table using Incorta SQL does create the parquet files.

If you need to do some complex manipulation ( build table A, query A and do some stuff - > table B, query table B and do some stuff -> C ... ) then it makes sense to use SQLi data source instead.

But memory usage will still be same. Only difference is, it will use memory in analytics service now and not loader service.