This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
Docs Learn Support
Help
Docs Learn Support
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

MV Failing to Load Due to Executor Lost Errors

poojapatel17
Employee
Employee

on ‎12-09-2025 03:47 PM

Introduction

A Materialized View (MV) fails during the transformation stage in Incorta. The error shows executors repeatedly being lost during Spark job execution.

Example Error Logs

ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] MV Name: "SalasBuzz_Data_App.HH_Transactions", Attempts: 1 Errors: 25/08/15 21:01:12 ERROR TaskSchedulerImpl: Lost executor 1 on 10.252.10.149: 
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 25/08/15 21:01:12 WARN TaskSetManager: Lost task 0.0 in stage 36.0 (TID 429) (10.252.10.149 executor 1): ExecutorLostFailure (executor 1 exited caused by one of the running tasks) Reason: 
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] Caused by: org.apache.spark.ExecutorDeadException: The relative remote executor(Id: 4), which maintains the block data to fetch is dead.
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 25/08/15 21:01:28 ERROR TaskSchedulerImpl: Lost executor 4 on 10.252.0.78: 
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 25/08/15 21:01:28 WARN TaskSetManager: Lost task 1.0 in stage 36.0 (TID 430) (10.252.0.78 executor 4): ExecutorLostFailure (executor 4 exited caused by one of the running tasks) Reason: 
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 25/08/15 21:01:30 ERROR TaskSchedulerImpl: Lost executor 2 on 10.252.13.191: 
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 25/08/15 21:01:30 WARN TaskSetManager: Lost task 2.0 in stage 36.2 (TID 446) (10.252.13.191 executor 2): ExecutorLostFailure (executor 2 exited caused by one of the running tasks) Reason: 
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 25/08/15 21:01:34 ERROR TaskSchedulerImpl: Lost executor 3 on 10.252.0.77: 
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 25/08/15 21:01:34 WARN TaskSetManager: Lost task 0.0 in stage 36.2 (TID 444) (10.252.0.77 executor 3): ExecutorLostFailure (executor 3 exited caused by one of the running tasks) Reason: 
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 25/08/15 21:01:41 ERROR TaskSchedulerImpl: Lost executor 10 on 10.252.79.56: 
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 	spark.kubernetes.driver.container.image=us-central1-docker.pkg.dev/incortacloud/sparkoverk8s/spark:v3.3-scala-executor
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 	spark.executor.extraJavaOptions=-XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 	spark.kubernetes.executor.volumes.hostPath.spark-local-dir-sdc.options.path=/mnt/disks/ssd1
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 	spark.dynamicAllocation.minExecutors=0
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 	spark.kubernetes.executor.secrets.spark-sa=*********(redacted)
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 	spark.kubernetes.executor.podTemplateFile=/home/incorta/scripts/spark-executor-pod-template.yml
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 	spark.kubernetes.container.image=us-central1-docker.pkg.dev/incortacloud/sparkoverk8s/spark-py:v3.3-py3-scala-r-executor
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 	spark.kubernetes.executor.volumes.hostPath.spark-local-dir-sdb.mount.path=/mnt/disks/ssd0
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 	spark.kubernetes.executor.volumes.hostPath.spark-local-dir-sdc.mount.path=/mnt/disks/ssd1
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 	spark.kubernetes.executor.container.image=us-central1-docker.pkg.dev/incortacloud/sparkoverk8s/spark:v3.3-scala-executor
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 	spark.kubernetes.executor.volumes.hostPath.spark-local-dir-sdb.options.path=/mnt/disks/ssd0
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 	spark.executor.cores=1
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 	spark.dynamicAllocation.executorIdleTimeout=500s
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 	spark.executorEnv.GOOGLE_APPLICATION_CREDENTIALS=*********(redacted)
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 	spark.kubernetes.executor.podNamePrefix=5ef9f81f-9503-4c0c-aa40-e7f7b5105ce2
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 	spark.executor.memory=2G [com.incorta.loader.transformer.SparkTransformer.execute]
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] Transformation error: INC_03070101: Transformation error 25/08/15 21:01:12 ERROR TaskSchedulerImpl: Lost executor 1 on 10.252.10.149: 
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 25/08/15 21:01:12 WARN TaskSetManager: Lost task 0.0 in stage 36.0 (TID 429) (10.252.10.149 executor 1): ExecutorLostFailure (executor 1 exited caused by one of the running tasks) Reason: 
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] Caused by: org.apache.spark.ExecutorDeadException: The relative remote executor(Id: 4), which maintains the block data to fetch is dead.
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 25/08/15 21:01:28 ERROR TaskSchedulerImpl: Lost executor 4 on 10.252.0.78: 
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 25/08/15 21:01:28 WARN TaskSetManager: Lost task 1.0 in stage 36.0 (TID 430) (10.252.0.78 executor 4): ExecutorLostFailure (executor 4 exited caused by one of the running tasks) Reason: 
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 25/08/15 21:01:30 ERROR TaskSchedulerImpl: Lost executor 2 on 10.252.13.191: 
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 25/08/15 21:01:30 WARN TaskSetManager: Lost task 2.0 in stage 36.2 (TID 446) (10.252.13.191 executor 2): ExecutorLostFailure (executor 2 exited caused by one of the running tasks) Reason: 
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 25/08/15 21:01:34 ERROR TaskSchedulerImpl: Lost executor 3 on 10.252.0.77: 
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 25/08/15 21:01:34 WARN TaskSetManager: Lost task 0.0 in stage 36.2 (TID 444) (10.252.0.77 executor 3): ExecutorLostFailure (executor 3 exited caused by one of the running tasks) Reason: 
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 25/08/15 21:01:41 ERROR TaskSchedulerImpl: Lost executor 10 on 10.252.79.56: 
ERROR: [00:01:45] [default | 81d7 | ETL | EXECUTION | RUNNING | SalasBuzz_Data_App.HH_Transactions-TRANSFORMATION-0 | FULL] 	spark.kubernetes.driver.container.image=us-central1-docker.pkg.dev/incortacloud/sparkoverk8s/spark:v3.3-scala-executor

Root Cause

This error typically occurs when Spark executors fail during shuffle or transformation stages. Large shuffle operations may overwhelm available executors, causing them to die or lose block data needed for downstream tasks.

Solution

Solution

To reduce shuffle overhead and rebalance tasks across fewer partitions:

  1. Edit the failing MV (in this case: SalasBuzz_Data_App.HH_Transactions.

  2. Add the following Spark properties:

     
    spark.sql.shuffle.partitions=10
    spark.executor.memory=2G

  3. Save and re-run the MV load.

These properties reduce the number of shuffle partitions, minimizing overhead and preventing executor loss during heavy transformation operations.

Related Material

For more details, see the Apache Spark Tuning and Monitoring Documentation

0 Kudos
Best Practices Index
Best Practices

Just here to browse knowledge? This might help!

Contributors
Version history
Last update:
‎09-10-2025 05:35 PM
Updated by:
Employee
Copyright © 2025 Khoros, LLC