0

Load table fail as Materialized View

I have a problem with Materialized View(using SQL not Python, I use INNER JOIN between two table which has 300M rows table and 2M rows table) when I try to load table, this is errors and can you help me how to fix it ?, thank you !

Schema: EXADATA_URIM
Start Time: 10:39:18 AM 08/14/2020
Error:
Table: UA_URIM_JCH
Message: INC_03070101: Failed to load data from [spark://BIPDB01:7077] with properties [[error, An error occurred while calling o50.parquet.
: org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.NoSuchMethodError: 'sun.misc.Cleaner sun.nio.ch.DirectBuffer.cleaner()'
    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at java.base/java.io.FilterInputStream.close(FilterInputStream.java:180)
    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    ... 9 more

Driver stacktrace:
    at org.apache.    at org.apache.    at org.apache.    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.    at org.apache.    at org.apache.    at scala.Option.foreach(Option.scala:274)
    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    ... 31 more
Caused by: org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    ... 1 more
Caused by: java.lang.NoSuchMethodError: 'sun.misc.Cleaner sun.nio.ch.DirectBuffer.cleaner()'
    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at java.base/java.io.FilterInputStream.close(FilterInputStream.java:180)
    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    ... 9 more

('Py4JJavaError', ':', Py4JJavaError(u'An error occurred while calling o50.parquet.\n', JavaObject id=o51))
]]
Error Trace:
com.incorta.api.exceptions.DataException: INC_03070101: Failed to load data from [spark://BIPDB01:7077] with properties [[error, An error occurred while calling o50.parquet.
: org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.NoSuchMethodError: 'sun.misc.Cleaner sun.nio.ch.DirectBuffer.cleaner()'
    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at java.base/java.io.FilterInputStream.close(FilterInputStream.java:180)
    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    ... 9 more

Driver stacktrace:
    at org.apache.    at org.apache.    at org.apache.    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.    at org.apache.    at org.apache.    at scala.Option.foreach(Option.scala:274)
    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    ... 31 more
Caused by: org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    ... 1 more
Caused by: java.lang.NoSuchMethodError: 'sun.misc.Cleaner sun.nio.ch.DirectBuffer.cleaner()'
    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at java.base/java.io.FilterInputStream.close(FilterInputStream.java:180)
    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    at org.apache.    ... 9 more

('Py4JJavaError', ':', Py4JJavaError(u'An error occurred while calling o50.parquet.\n', JavaObject id=o51))
]]
    at com.incorta.loader.transformer.SparkTransformer.execute(SparkTransformer.java:94)
    at com.incorta.loader.transformer.SparkTransformer.execute(SparkTransformer.java:41)
    at com.incorta.loader.transformer.TransformerJob.transform(TransformerJob.java:92)
    at com.incorta.loader.transformer.TransformerJob.transform(TransformerJob.java:63)
    at com.incorta.loader.etl.TransformationPhase.transformTable(TransformationPhase.java:147)
    at com.incorta.loader.etl.TransformationPhase.access$000(TransformationPhase.java:33)
    at com.incorta.loader.etl.TransformationPhase$1.doAction(TransformationPhase.java:89)
    at com.incorta.util.ConcurrentService$3.call(ConcurrentService.java:154)
    at com.incorta.util.ConcurrentService$3.call(ConcurrentService.java:142)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at com.incorta.util.ConcurrentService$Job.execute(ConcurrentService.java:321)
    at com.incorta.loader.etl.TransformationPhase.executePhase(TransformationPhase.java:96)
    at com.incorta.loader.etl.ETLJob.transform(ETLJob.java:267)
    at com.incorta.loader.etl.ETLJob.execute(ETLJob.java:102)
    at com.incorta.loader.LoaderImpl.doLoadData(LoaderImpl.java:275)
    at com.incorta.loader.LoaderImpl$1.call(LoaderImpl.java:206)
    at com.incorta.loader.LoaderImpl$1.call(LoaderImpl.java:203)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
2replies Oldest first
  • Oldest first
  • Newest first
  • Active threads
  • Popular
  • Hi Nopu Nguyen a couple of tips for this: 1) This is likely a memory issue where your default memory allocation to the spark executor (1GB) is not sufficient to process 300M records. 2) Spark is attempting to use an API method due to the tight memory but if you're using OpenJDK 11 it does not support the method as revealed by the below error:

    java.lang.NoSuchMethodError: 'sun.misc.Cleaner sun.nio.ch.DirectBuffer.cleaner()'

    If your Spark instance has sufficient available memory, allocate more memory to the MV and increase until the MV completes successfully. You can override the default by adding properties to the MV itself:

    Another option is to install OpenJDK 8 on your server and use it to run Spark. This will allow you to run MVs with less physical memory as OpenJDK 8 supports sun.nio.ch.DirectBuffer.cleaner(). After installing OpenJDK 8, follow the steps below. 

    1. Login to the backend and navigate to the following path
    <Incorta_Installation>/IncortaNode/
    2. Edit the file startSpark.sh
    Add the following lines

    export JAVA_HOME="<open-jdk8 Path>"
    export JRE_HOME=$JAVA_HOME/jre
    

    3. Restart spark
    4. Check that the spark is started with openJDK8 using the following command

    ps -ef | grep spark
    Like 2
      • Nopu Nguyen
      • Human
      • Nopu_Nguyen
      • 3 mths ago
      • Reported - view

      Dustin Basil I try to set sprak.executor.memory and maybe it works fine. thank you so much.

      Like
Like Follow
  • Status Answered
  • 3 mths agoLast active
  • 2Replies
  • 57Views
  • 2 Following

Product Announcement

Incorta 4.9 is now Generally Available (GA)!!!