topic Re: How to perform incremental load for particular table inside schema without performing Full load to the other tables inside schema. in Data & Schema Discussions

How to perform incremental load for particular table inside schema without performing Full load to the other tables inside schema.

Madhan — Tue, 09 May 2023 06:42:16 GMT

Please check the screenshot for reference.

Thanks,

Madhan

Re: How to perform incremental load for particular table inside schema without performing Full load to the other tables inside schema.

VenkateshP — Tue, 09 May 2023 11:13:37 GMT

Hi Madan,

In incorta if you do incremental at the schema level, it will consider incremental load only for which
incrementals have been enabled and the remaining tables will be considered full load by default.

However if you have access to on premise physical server, then you can do incremental for a single table in the schema using below steps.
Steps:-
i)The CLI for Incorta is available on an Incorta node with the Analytics Service running and the CLI python file is incorta.py.
The default location of:
/<IINCORTA_INSTALLATION_PATH>/IncortaNode/bin/incorta.py

To use the CLI, you must first establish a web user session. Typically, you store the session credentials in a local shell
variable. To create a session, use the CLI login command as below.

The login command requires the following arguments:
session=`python incorta.py login http://<url>:8080/incorta tenant <username> <password>`

In above case <username> is username , <password> is password , <tenant> is respective tenant and <url> is login URL of the Analytics Service

ii)To load incremental for a single table or schema use the method of 'load_schema_incremental'

for eg., to load only 'Customers' table in 'SALES' schema

python incorta.py load_schema_incremental $session SALES Customers

iii)logout of the session i.e. closing the opened session

python incorta.py logout $session

For reference ,please follow below url.

https://docs.incorta.com/5.1/tools-cli

Thanks

Venkatesh

Re: How to perform incremental load for particular table inside schema without performing Full load to the other tables inside schema.

vengat — Tue, 15 Aug 2023 20:03:04 GMT

is the load_schema_incremental allow TABLE as arg?. Because I dont see that in CLI.

Re: How to perform incremental load for particular table inside schema without performing Full load to the other tables inside schema.

dylanwan — Wed, 31 Jan 2024 19:17:38 GMT

If you want any table to not fetch any data during scheduled or ad hoc incremental refresh, you can add a dummy incremental logic to the table.

For example, add the filter that returns nothing in the SQL based table.

SELECT COL1 , COL2 FROM SCHEMA1.TABLE1 WHERE 1=2