Data Load through API Calls

[shashidhar_sris]

Hello All,

I need to work on an MV using spark python where I need to call around or more than 1000 APIs using an endpoint. I can loop it 1000 times by passing different parameter with same endpoint.

Is there any restrictions from Incorta if we hit the source multiple times and will it take more time to load the data?

Please suggest.

Regards,

Shashidhar.S

[JoeM]

@shashidhar_sris If you are reading data into Incorta, the API limits would be enforced from the source system and not through Incorta. I recommend checking your source systems to see if there is a rate limit. 
Will it take more time? It depends on the source. More often than not, bulk importing through a JDBC driver will be more efficient (if available), but an API can be better if writing specific extraction logic. 

[shashidhar_sris]

Hi @JoeM ,

Actually we have restriction from source side and max response we can get is 10MB only. But within this limit we would like to write an effiecient code to not hit API multiple times.

Is there any best practice that we can adopt for better performance?

Regards,

Shashidhar.S