Data Loading Approach For Datalake

Integrator.io
, , , ,
1

I'm looking for for best practices for loading data from one source (HubSpot, Salesforce, etc.) to a data lake (AWS S3 in our case). My thought is that for the initial data load I would select the export type of "All - always export all data" and then once that flow has completed switch the export type to "Delta - export only modified data". Will that approach work or is there an alternative method that would be better?

5301043caa20740200000001-9964d90f0fcc426f8013744aa6c767e5.png
5301043caa20740200000001-9964d90f0fcc426f8013744aa6c767e5.png1248×370 14.8 KB

5301043caa20740200000001-717ab6d2378a4c448838cc7099565310.png
5301043caa20740200000001-717ab6d2378a4c448838cc7099565310.png1208×288 14.1 KB

2

Yes, the best practice is to use 'All' for your initial mass loads, and then switch it over to 'Delta' when you are done.

3

Perfect. Thanks so much Scott!