1

What are the types of data load that Incorta supports?

5replies Oldest first
  • Oldest first
  • Newest first
  • Active threads
  • Popular
  • Incorta Supports 4 types of data load:
    1-Full Load from the datasource.
    2-Incremental load which load only the changes happened from the pervious load.
    3-load from staging which load data from existing parquet files.
    4-load from snapshots which load data from snapshots written on disk.

    Reply Like
  • Thanks. How do we load just to Parquet? Please explain that as well.

    Reply Like 1
    • Mateen Mohammed 

      There are two steps to the Incorta data load: 1) Extract data via connector to Incorta's Base Layer (aka. Parquet files) and 2) Load data from Parquet files into Incorta's Direct Data Mapping engine.

      If you mean #1:
      First we extract the data from a big variety of datasources that Incorta supports into Incorta schemas then Incorta will write those data in parquet files on disk (in the installation folder of Incorta) as columnar data files.

      If you mean #2:
      load from staging (parquet) is very useful for example if the data is extracted from datasource and an error happens in data loading in that case we use load from staging  which will skip the extract phase and starts loading from staging direct. This can be done by choosing load from staging  option inside the schema view as show in the attached image
       

      Reply Like
  • For better understanding for the load phase..load phase has some steps until data loaded in memory so let's make them easy and to the point:
    1-Data extracted from datasources into our staging area(parquet).
    2-Incorta loader read data from our staging area(parquet) and starts loading them into engine.

    Snapshots:
    Incorta uses snapshots files to save its state on disk and if the system get down for any reason you can return the last Incorta state by loading from snapshot and this helps a lot to guarantee the persistence of your data.

    Reply Like
    • Mohamed Orikat  thanks for your information, but I have some questions.

      1. when I'm trying to load my schema with snapshot load, its not getting loaded. Will you please explain.? Should we make any changes to load snapshot data?

      2. As snapshot load is not working I did incremental load and after that when I'm trying to make report in schema its showing only one data against every snapshot_date. Its not getting previous as well as todays data, only showing one data against every column in report.

      Please help.

      Thanks

      Reply Like
Like1 Follow
  • Status Answered
  • 1 Likes
  • 1 yr agoLast active
  • 5Replies
  • 566Views
  • 3 Following