Looking for ways to Bulk load about 6M records into a customer stage table ( no scripts, straight loads) using Celigo bulk APIs + 24 concurrent threads. From our current test, looks for a small load of 10K records, ONLY 1 thread is being utilized and is taking about 17 min, and we are looking at ways to force Celigo to use ALL 24 threads, so as to get a a significant uptick in data load throughput
From Netsuite side, the Integration is set to 24 concurrent web service connection. Please advise if there a setting that can be tweaked, or a pattern to build the flow needs to be utilized.
@johnpapavizas, it sounds like you have concurrency for the integration set within NetSuite, but it looks like you don't have it set on the connection settings within Celigo. Here are a couple of good articles to follow that explain how concurrency settings and queing work within Celigo:
https://docs.celigo.com/hc/en-us/articles/360034522472-Govern-concurrency-of-NetSuite-connections
https://docs.celigo.com/hc/en-us/articles/360043926372-Configure-connections-to-optimize-throughput-and-governance
Additionally, you may need to modify batch sizing so that more records get exported per page. Here is a good article about that.
https://docs.celigo.com/hc/en-us/articles/360043927292-Fine-tune-integrator-io-for-optimal-data-throughput#page-size