topic Re: What exactly does this do? >> Please, increase max retries threshold property. in Data & Schema Discussions

What exactly does this do? >> Please, increase max retries threshold property.

RADSr — Fri, 18 Nov 2022 15:28:42 GMT

I have a table which loads from a DB view. It takes a long time ( v. very quick for incremental, so hurrah for CDC! ) and just recently ( i.e. yesterday ) started erroring out when doing the full load. The error message includes this:

The data source is taking too long to process [FETCH_NEXT_RECORD_SET] request with guid 
[fe9f21dc-8e51-476a-8701-700b7cecc00f] . 

Please, increase max retries threshold property. Doubling the value should suffice. 

You can add/update the property `data.agent.command.max.retries` in `service.properties` current value (10).

which ... OK. But before I go futzing around w/ settings I want to make sure I know what they are doing. Is this really just increasing max retries *with the same timeout* which seems like it would put more strain on the source without increasing my probability of getting rows back or is it increasing the timeout value for the initial query?

Further background - this view behavior is expected. When it was running full loads successfully it would start the same time as other tables, but wouldn't show any rows returned for a long time. So the timeout makes sense and the view is not mine so I can't control it from that side.

I did the search bar here support, and documentation but no luck going.

Any insight?

Re: What exactly does this do? >> Please, increase max retries threshold property.

JeffW — Fri, 18 Nov 2022 17:02:03 GMT

Randy, Is this a Cloud implementation? This i've seen before quite often from the Data Agent. The error is exactly what it says, the Data Agent has sent a request for a query to the DB source and it is waiting for a response. There is a setting for how many retries it will make. I think by default it might be set to 10 and when we increase it support normally bumps it up to 25. However, as you've indicated the root cause is really that the source is not returning rows. You might want to determine why the source is taking so long to respond. Are they throttling the connection? It does not appear to be dropping the connection otherwise you'd get an entirely different error. Need to understand why the SQL is taking so long to return rows when DA is essentially waiting until eventually it times out.

Re: What exactly does this do? >> Please, increase max retries threshold property.

RADSr — Fri, 18 Nov 2022 17:18:29 GMT

Yes - cloud implementation.

Unfortunately I don't have any insight or access to the source DB view -- I know it's pulling "live" data for a lot of transactional tables so there's probably a bunch of caching and table locking going on under the hood, but I don't get any of the cool fun access to know that directly.

What I really need then is a way to increase the timeout from Incorta waiting for rows to be returned. Do you know if there's a configurable property for that?

In the interim I'm going to have to fiddle around w/ incremental loading a couple months at a time until I'm caught up to the present.

Re: What exactly does this do? >> Please, increase max retries threshold property.

amit_kothari — Sat, 19 Nov 2022 17:26:24 GMT

Randy - what is the source database?

Re: What exactly does this do? >> Please, increase max retries threshold property.

RADSr — Sat, 19 Nov 2022 19:30:05 GMT

SQLAnywhere.

Re: What exactly does this do? >> Please, increase max retries threshold property.

amit_kothari — Mon, 21 Nov 2022 01:01:26 GMT

You can define extraction timeout at table level . Check screenshot.

Re: What exactly does this do? >> Please, increase max retries threshold property.

RADSr — Mon, 21 Nov 2022 15:30:50 GMT

I looked in the docs here and searched for "timeout" and "extraction" "extraction" has some hits, but not a timeout setting that I can see.

Re: What exactly does this do? >> Please, increase max retries threshold property.

JeffW — Mon, 21 Nov 2022 21:30:47 GMT

Randy, the error message itself says it is the Data Agent that is timing our while it waits for a reply from the Data Source. Therefore there IS a max retries configuration that we can update, and support knows how to do this. That being said, there is some reason that the particular query is taking so long to return to the Data Agent after it is submitted to the data source. I would recommend having the customer assist in monitoring the incoming query request and then determine why it is running so long. I had a very similar issue recently on a query from ERP ACDOCA ledger table. It's a large table and bumping up the retries assisted in allowing the query to finish, but only because the Data Agent was able to wait longer for the results.

Re: What exactly does this do? >> Please, increase max retries threshold property.

marcpaige — Mon, 22 May 2023 18:18:30 GMT

I am interested in this issue as well. I have a very large table that is experiencing this same error. @JeffW, please confirm that I need to open a ticket with support to make this alteration.