0

Data File Load oddity

I have a standard CSV file that I loaded the first time without issue. I created a new schema and table to house the data from the file. I added a join to the table and went to explore and the join didn't show what I expected so I reloaded the data file based table. This time the row count dropped to 1 from the original 483. I performed a full load. Please help me understand what I did wrong.

10replies Oldest first
  • Oldest first
  • Newest first
  • Active threads
  • Popular
  • What do the load details say?   Does it show rejected records?   If records were rejected it should have a link to an error to help indicate why. 

    Like
    •  Dan Brock , I have pasted a screenshot of the full load. As you can see, there are no rejected records. The second screenshot shows the record count from the schema.

      Like
    • Marc Paige What is the key for the table set to?  Is it possible that you're actually INSERT'ing the first record and then updating that record 482 times because all rows share the same (non-unique) primary key?

      Like
    • Dan Brock The key is set to the date on the records. I have tried it with the date set as a key and not as the key and I still only get 1 row on a full load.

      I need to load files like this one every day in a snapshot way. I am going to experiment with that, but I could not find any documentation on a method for snapshotting data files.

      Like
    • Marc Paige You may want to load to a non-optimized table first and then build an Incorta table subsequently that loads the date as part of the key.  If you have a snapshot table, your PK will need to be a combination of your unique key and the timestamp.  Make sure you're loading the PK as a timestamp and not just a date.  It might see all values as the same with a date which is why you're only getting 1 record.

      Like
    • Dan Brock Turning off optimization and then full load worked to get all of the rows back. Thanks for that.

       

      I need more guidance on the snapshot information. Is there a specific way to load data files (CSV with only a date and not true key) into daily snapshots?

      Like
    • Marc Paige Check out this post https://community.incorta.com/t/361jr6/creating-snapshot-tables-with-incorta

       

      It may not answer all of your questions.  Feel free to reach out to me directly with any specifics.

      Like
    • Dan Brock I read this already and did not find help wrt data files.

      Like
  • The solution to this problem is to use a data folder and union the files in the table definition. Ahmed from customer support provided the basic steps and Siva added the details. 

    I think an instructional video showing this technique would be very helpful.

    Like
  • Marc Paige  Thanks for your feedback. we will look into it. 

    Like
Like Follow
  • Status Answered
  • 9 days agoLast active
  • 10Replies
  • 22Views
  • 3 Following