Issue Backing Up Files in S3 Export

Integrator.io
1

Hi all,

I’m trying to configure a flow that uses a specific Amazon S3 bucket to retrieve .json files in a subfolder, and then move (back up) those files to a separate folder after they are processed. In the settings I define:

  • My Bucket name (let’s call it “my-bucket” here)
  • The path to the S3 folder containing the files I want to process as Prefix
  • The path where I want the files to be moved after they are processed as Backup bucket name

When I test the flow, Integrator does retrieve the file(s) for processing, but once it’s done it moves the files in a way that I didn’t expect.

  • Source: s3://my-bucket/partners/corvelay/DEV/INBOUND/purchaseorders/NEW/
  • Expected destination: s3://my-bucket/partners/corvelay/DEV/INBOUND/purchaseorders/PROCESSED/
  • Actual destination: s3://my-bucket/partners/corvelay/DEV/INBOUND/purchaseorders/PROCESSED/partners/corvelay/DEV/INBOUND/purchaseorders/NEW/

Why does the export join the source bucket and prefix with the backup prefix? How can I get it to move the files cleanly to the S3 folder of my choice without moving the entire directory?

image
image2134×1914 307 KB

2

It's because that's only the "backup bucket name," so we take your file key (partners/corvelay/etc/file.json), which is everything after the bucket name, and simply move that to the other bucket (backup bucket name + retrieved file key). It's currently not possible to customize the backup file key to do what you're wanting. You could make another bucket, such as my-bucket-processed, or move to a pattern like my-bucket/PROCESSED.

I'll also copy your post to Product Feedback since this would be an enhancement request for custom backup file keys.

3

A post was split to a new topic: Support custom file key backup for S3

4

Thank you for clarifying and moving to the product feedback forum! I’ll add some more detail there about my use case and why this feature would be useful.

1 Like