Before anyone types "it depends" know that I'm a consultant and claim that answer for myself 😉
I am interested to know if there are any measurements which can help me determine a strategy for doing incremental builds as my routine, and only doing full loads as runtime performance dictates. I'm predominantly on Incorta cloud so kind of don't care if I clutter up a directory w/ 1 or 1000 files, but do care if I watch the "wheel spin" when I open dashboards.
I don't know enough to know if this should be based on number of parquet files, size of the files, a combination of both, or some other factor(s).
Hoping the fine folks in product development, support, or services have done some benchmarking and can provide some guidance.
.svg "Partner"){kind=icon}
