We have a Company Sync flow from Shopify to NetSuite. If we use export type "Delta", we are missing records from Shopify, hence we have set the flow to export type "All". Currently we have 1200 records to process but I want it to optimise in such a way that it doesn't load much on Celigo and NetSuite.
Creating two different imports for add and update will it be helpful?
What approach do you think we should use to make it effecient and optimised?
Hi Lakhan,
Your use case is very common and I do have some thoughts on doing this optimally.
You should be able to fetch all records still using the timestamp of Export type "Delta" (lastExportDateTime) and filter based on the last modified timestamp of the Shopify company i.e. updatedAt > lastExportDateTime.
On your initial run, you can manually run it and set the lastExportDateTime back very far, i.e. 2000 such that the initial run fetches all the records after that date. Moving forward, it will only fetch the records with newer "updatedAt" timestamp. On Shopify company record creation, the "updatedAt" timestamp should default to the same as the "createdAt" timestamp so the initial run should get the all company records in your Shopify system as long as you set the timestamp far back enough.
As for the imports, though the NetSuite adaptor does support combining and managing both ADD/UPDATE in a single step, I personally prefer separating them into two imports because it makes troubleshooting and maintaining them easier, particularly with mappings. For example, on ADD record there may be a lot of required fields, but on UPDATE you may only want to update only a few fields.
You'll just want to make sure you set the order and filter properly, i.e. something like:
- Export
- Update (ignore missing records, setup criteria on how to identfy existing)
- Add (ignore existing records)
If you add before update, the update operation will always be performed so it's a slight waste of bandwidth.
Hope this helps!
Simon
Hey Simon,
Thanks for your response. As I have mentioned that while doing it as delta flow based on updatedAt, the flow is missing records. What else should be compiled with this setting to make sure the flow doesn't miss records?
Also, onto the point about different steps, yes fewer fields need to be updated in record than fields while creation of record.
Thank you!
Hi Lakhan,
Got it, that's unusual that the flow would miss records when using updatedAt filter. Could you elaborate on how or what records are being missed when updatedAt is used? It should be the only filter in place, and to grab ALL existing records on the first sync you can set the initial timestamp back very far, i.e. year 2000.
If there's no way for you to filter properly based on updatedAt timestamp, then export type "All" might be the only option, and a periodic syncing of all 1200+ records. This would use a lot more bandwidth, but still well within the volume capabilities of the flow (performance will depend on your NetSuite instance and concurrency settings).