Composite Import not honoring Concurrency ID Lock Template

[Sean_Ayotte]

Brand new to Celigo, so apologies if this is rudimentary.

In a Composite import step (Create new records & ignore existing records), I am performing a dynamic lookup to determine whether the record exists before sending the request to add the record. Multiple records may exist on the page, so I am using the Concurrency ID Lock Template field to prevent passing duplicates through the lookup and POST steps of the import at the same time. The duplicates are still passing through the lookup in parallel without the subsequent record(s) pausing to allow the first to get created. What am I missing?

Error log excerpt:

{"url":"https://api.safetyculture.io/directory/v1/folder","method":"POST","body":"{\"name\":\"XYZ\",\"meta_label\":\"state\"}","headers":{"content-type":"application/json","sc-integration-id":"sc-readme","Authorization":"Bearer ********","accept":"application/json"},"requestIndex":0}


05/09/2025 11:40:37 am efdc94d8973a410fa1eda3b3e28f1c9b import 681e112852d5f629d8bb7983
{"headers":{"date":"Fri, 09 May 2025 15:40:37 GMT","content-type":"application/json","transfer-encoding":"chunked","connection":"keep-alive","server":"cloudflare","cf-ray":"93d24bdf29a13b2f-IAD","grpc-metadata-content-type":"application/grpc","grpc-metadata-trailer":"Grpc-Status, Grpc-Message, Grpc-Status-Details-Bin","vary":"Accept-Encoding","x-ratelimit-limit":"800, 800;w=60","x-ratelimit-remaining":"793","x-ratelimit-reset":"23","strict-transport-security":"max-age=15552000; includeSubDomains; preload","x-frame-options":"deny","x-xss-protection":"1; mode=block","x-content-type-options":"nosniff","content-security-policy":"block-all-mixed-content","x-permitted-cross-domain-policies":"none","referrer-policy":"origin","expect-ct":"max-age=86400,enforce","access-control-allow-credentials":"true","sc-ratelimit-bucket":"s12.directory.v1.Directory","cf-cache-status":"DYNAMIC","alt-svc":"h3=\":443\"; ma=86400","authorization":"Bearer ********"},"body":"{\n  \"folder\": {\n    \"id\": \"8c311f45-a1f5-4389-9d2c-a83e2b4bc4ec\",\n    \"name\": \"XYZ\",\n    \"org_id\": \"a4be65b8-3dba-4e6e-9f23-332bcae06a37\",\n    \"creator_id\": \"f4edfa5a-8faa-432f-8fe6-08112fe91783\",\n    \"created_at\": \"2025-05-09T15:40:37.678055297Z\",\n    \"modified_at\": \"2025-05-09T15:40:37.678055297Z\",\n    \"meta_label\": \"state\",\n    \"members_count\": 0,\n    \"deleted\": false,\n    \"managers_count\": 0\n  }\n}"}


05/09/2025 11:40:37 am 7b819d4126a64ffa92eb42a08a35559f import 681e112852d5f629d8bb7983
{"url":"https://api.safetyculture.io/directory/v1/folder","method":"POST","body":"{\"name\":\"XYZ\",\"meta_label\":\"state\"}","headers":{"content-type":"application/json","sc-integration-id":"sc-readme","Authorization":"Bearer ********","accept":"application/json"},"requestIndex":0}


05/09/2025 11:40:38 am 7b819d4126a64ffa92eb42a08a35559f import 681e112852d5f629d8bb7983
{"headers":{"date":"Fri, 09 May 2025 15:40:38 GMT","content-type":"application/json","transfer-encoding":"chunked","connection":"keep-alive","server":"cloudflare","cf-ray":"93d24be07b1a3b2f-IAD","grpc-metadata-content-type":"application/grpc","grpc-metadata-trailer":"Grpc-Status, Grpc-Message, Grpc-Status-Details-Bin","vary":"Accept-Encoding","x-ratelimit-limit":"800, 800;w=60","x-ratelimit-remaining":"792","x-ratelimit-reset":"23","strict-transport-security":"max-age=15552000; includeSubDomains; preload","x-frame-options":"deny","x-xss-protection":"1; mode=block","x-content-type-options":"nosniff","content-security-policy":"block-all-mixed-content","x-permitted-cross-domain-policies":"none","referrer-policy":"origin","expect-ct":"max-age=86400,enforce","access-control-allow-credentials":"true","sc-ratelimit-bucket":"s12.directory.v1.Directory","cf-cache-status":"DYNAMIC","alt-svc":"h3=\":443\"; ma=86400","authorization":"Bearer ********"},"body":"{\n  \"code\": 6,\n  \"message\": \"Folder with name \\\"XYZ\\\" already exist in parent/root folder\",\n  \"details\": []\n}"}
type or paste code here

Perhaps unrelated and maybe I am misunderstanding this, but shouldn't the "Action to take if unique match not found", just be "treat as new" or something? It is supposed to be checking if there is an existing record, and if there isn't one, what should it do, but that is answered by Composite type (Create new records & ignore existing records) ... or is this looking for something different?

[tylerlamparter]

I actually have a very similar setup, but the composite setup doesn't work like you're thinking. The composite setup will first perform all of the lookups, then perform all of the imports based on the result of the lookup. It won't do a lookup on the first record in the page, then perform the import, then start the lookup for the next record. I could see how this ordering could be useful, though. Also, it is technically still respecting your concurrency lock because it would be performing the import part in series. Here is a debug log of my very similar setup.

You could achieve what you're wanting by setting the page size on your export to 1 and then setting the concurrency of the connection on your import step to 1 as well. This would ensure it's all going in FIFO.

As for the "Action to take if unique match not found," it applies when there is more than one result returned from your lookup. If this occurs, you can either set the resource identifier path to the 0 index (if you're okay with it just choosing the first), or you can have it fail out, or have it return null, which should trigger the create operation (I assume, as I haven't tried this).

[Sean_Ayotte]

Thank you. I had it set at Page size=1, but I'm guessing that was a different iteration.

then setting the concurrency of the connection on your import step to 1

How do I do this? I have the option of a lock template, not a numeric. I have mine set to what would be my "group by" or "distinct" value if I was doing this in SQL ...

[tylerlamparter]

If you go to your import>edit connection>HTTP toggle at the top>advanced>concurrency/target concurrency.

[Sean_Ayotte]

Thank you. My query is now held up by an issue with an endpoint provider, so I'm not 100% if my issues is resolved, but I'll keep pushing.