Showing results for 
Search instead for 
Did you mean: 

Incremental loads - when and how much does it negatively impact runtime queries?


Before anyone types "it depends" know that I'm a consultant and claim that answer for myself  😉 

I am interested to know if there are any measurements which can help me determine a strategy for doing incremental builds as my routine, and only doing full loads as runtime performance dictates.  I'm predominantly on Incorta cloud so kind of don't care if I clutter up a directory w/ 1 or 1000 files, but do care if I watch the "wheel spin" when I open dashboards.

I don't know enough to know if this should be based on number of parquet files, size of the files, a combination of both, or some other factor(s).  

Hoping the fine folks in product development, support, or services have done some benchmarking and can provide some guidance.