0

Is there a way to manually compact a table?

I have a large table (85m rows) that I'm using as a source for a MV (python) and I was having an issue getting any rows returned. I know the mv works because it's working in another environment. The only thing I've been able to find is when I try to preview the data for the source table I get a message that reads "INC_05022201: Table [RTXN] is not compacted yet."

From what I've found on the Incorta documents is that if you have Enable Always Compact turned on in the tenant settings that it should trigger compaction immediately after a successful load. The table has loaded successfully for a few days now but I'm still getting that message.

Is there a way to manually compact a table?

3replies Oldest first
  • Oldest first
  • Newest first
  • Active threads
  • Popular
  • I am afraid that there is no way to trigger compaction manually. 

    After you enable Aways Compact option in the tenant setting, did you run a full load of the schemas?   Did you ever restart incorta laoder services?  It may trigger compaction during start up.

    If the MV exists in the same schema as the source table, and you have the Always Compact option off,  when you run a regular incremental load on that schema,  before MV is being executed, incorta will try to run the compaction process for all source tables involved within the same schema.

    If you are using SQLi option against SQLapp, please turn Always Compact on.  If most of tables are not used from SQLi and are only used in MVs within the same schema, you can turn the Always Compact off.  When you turn Always Compact on, the compaction process will be triggered right after the extraction of the table finishes and start when there are enough resource available.  Compaction is an asynchronous process and can be executed in parallel with the rest of extraction and load job.  

    To check if compaction run successfully, please check <tenant folder>/compact/<Schema Name>/, you should see files with the table name there, by comparing the size of those files, we will verify if the data are compacted.  

    Like
  • Thanks for the information.

    We've had Always Compact turned on for as long as I know. The schema was new (imported from our Dev environment) so after it was imported I ran a full load. The loader has been restarted at least twice this week. This schema loads incrementally daily.

    I checked the compacted folder and there are 12 tables in this particular schema, 7 of them show a file size and they work as expected when I preview the data (that's when I see the not compacted error) and the other 5 all show a file size of 0. For the specific table I'm working with it shows RTXN.0 and within that folder there is one file named part-000000 with a size of 881, created on the day I first ran the full load.

    Like
  • Tried a few more things and I ended up having to do a full load on just that one table. I'm now able to preview the data and the mv is working as expected.

    Like
Like Follow
  • Status Answered
  • 6 mths agoLast active
  • 3Replies
  • 28Views
  • 2 Following

Product Announcement

Incorta 4.9 is now Generally Available (GA)!!!