topic Re: Guidelines for SPARK property settings? in Administrative Discussions

Guidelines for SPARK property settings?

RADSr — Tue, 10 Oct 2023 17:49:36 GMT

I've got an MV using spark sql running for ~ 1.5 hours ( and counting ) - the same query ran in under 4 minutes using an Incorta SQL derived table.

For prior issues support has recommended adding some properties to the datasource screen.

Is there a guide as to when to use which? It would be absolutely swell to be able to translate this ( screencap from drilling down in Spark /applications page ) into an informed action.

* Incorta UI still shows this as "enriching." Candidly I don't know whether all those "FAILED" messages are telling me to wait it out while it retries, or to give it up because something essential has already gone wrong...

Re: Guidelines for SPARK property settings?

RADSr — Tue, 10 Oct 2023 19:34:39 GMT

I killed that load after about 2 hours, applied some settings from an earlier ( different MV ) support ticket and it went down to 1 minute 24 seconds.

I'm thinking I should just default to those settings for everything...

Re: Guidelines for SPARK property settings?

dylanwan — Tue, 24 Oct 2023 12:57:39 GMT

Incorta MV assistant feature was created for this purpose.

Those failed messages mean that the Spark cluster keeps trying to rerun a task but failed may be due to resource issue.

Re: Guidelines for SPARK property settings?

dylanwan — Tue, 24 Oct 2023 13:00:39 GMT

The default values can be configured from the CMC.