We have a requirement to connect to a SQL DB and export any new records, so that we can do some basic mapping and then upload the into Netsuite.
We have around 15k records new every run which in some cases will exceed the file limit 5mb and the flow will error out. In order to avoid these errors we would like to limit the number of records and be able to re run the process flow, to bring the remaining data set. The challenge is that the created date record will be the same for the 15k records.
Is there any work around where Celigo mimics the pagination of an API with a SQL connection? Or any other ideas