File handling generates more repeat questions than almost any other topic on this forum. The questions look different on the surface -- Gmail attachments, presigned S3 URLs, multipart uploads, PDFs stuck in a RESTlet response -- but nearly all of them trace back to one mental model. This post lays out that model once, then gives you the solution for each common use case. If your scenario isn't covered, reply below and we'll add it.
The one concept: records vs. files
integrator.io moves data through a flow in one of two forms:
-
Records are parsed, structured data. They're what you map, filter, and transform. Each record is capped at 5 MB per flow step.
-
Files (blobs) are raw bytes: PDFs, images, ZIPs, or any file you don't want parsed into records. File bytes never ride inside a record. When a flow step fetches a file, integrator.io streams the bytes into its internal file storage and hands your flow a
blobKey-- a pointer to those bytes. Files moved this way have no practical size limit.
When a step fetches a file, the record that comes out of it looks like this:
{
"success": true,
"fileMeta": {
"fileName": "invoice-1042.pdf",
"lastModifiedTime": 1636586398000
},
"blobKey": "717a84e9ac4249268b8b352018d1af72"
}
Four rules follow from this, and they answer most file questions on their own:
-
Fetching a file means producing a
blobKey. If you don't have ablobKey, you don't have a file -- you have data inside a record. -
Sending a file means referencing a
blobKey. Every file-sending step has a Blob key path field under Advanced settings that tells it where on the record to find the pointer. If you're transferring a blob, this field is required -- and its syntax is plain dot notation relative to the record root:blobKey, ornested.blobKeyif you response-mapped it deeper. Notrecord.blobKey, and not JSONPath. -
A
blobKeyonly lives as long as the flow run that created it. You can't pass one to a different flow. If a file needs to cross flows, stage it somewhere real -- see use case 5. -
Keep file bytes out of the record lane. Carrying a base64 string inside a record to move a file runs you into the 5 MB record cap and gives you nothing a downstream transfer step can use. Use case 3 shows the right way.
Where this shows up in Flow Builder
| Position | Step type | Lane | What you get or need |
|---|---|---|---|
| Source | Export records from source application | Records | Parsed records |
| Source | Transfer files out of source application | Files | A blobKey per file |
| Middle | Look up additional records (per record) | Records | Data merged onto your records |
| Middle | Look up additional files (per record) | Files | A blobKey per record, via response mapping |
| Destination | Import records into destination application | Records | Field mappings |
| Destination | Transfer files into destination application | Files | Requires Blob key path |
File providers -- FTP/SFTP, Amazon S3, Google Drive, Box, Dropbox, Azure Blob Storage, the NetSuite file cabinet, Celigo Storage, and the like -- offer the transfer step types everywhere, including as the flow's source, because they can list a folder and pick up whatever is in it. A generic HTTP API can't be listed like a folder, so an HTTP flow can never start with a file transfer. It has to start with a record export (a list of messages, documents, or attachments), and the files come in per record after that. That one constraint explains a large share of the confusion out there.
Find your use case
| You need to... | Use case |
|---|---|
| Move files from one file provider to another (FTP to S3, Drive to NetSuite...) | 1 |
| Get files from an API that returns URLs to the files | 2 |
| Get files from an API that returns file content in the response body (raw or base64) | 3 |
| Send a file to an HTTP API (raw body, multipart/form-data, or something non-standard) | 4 |
| Turn records into a file, then send that file to an HTTP API | 5 |
| Send base64 inside a JSON body because the API won't take raw files | 6 |
Use case 1: move files between file providers
The straightforward one, and the template every other case funnels into.
-
Source: Transfer files out of source application. If you're on an export form instead, the equivalent is choosing not to parse the files. Either way, each file becomes a record shaped like the JSON above, with a
blobKey. -
Destination: Transfer files into destination application. Don't generate files from records -- you're passing an existing file through. Set Blob key path to
blobKeyunder Advanced settings.
That's the whole flow. If you want the file's contents as records instead (a CSV to parse, for example), that's not a transfer at all -- use an export and configure parsing, and you'll get records rather than a blobKey.
Use case 2: the API returns file URLs
Common with presigned S3 links, attachment URLs, and report-generation endpoints.
-
Start the flow with a record export -- the list of orders, messages, or documents whose responses contain the URLs.
-
On that export (or on a record lookup mid-flow), open Non-standard API response patterns and set Path to file URLs in HTTP response body to the field holding the URL. It accepts JSON path notation, so
$.attachments[*].urlwalks an array and downloads every file; you can also comma-separate multiple fields. -
If the URLs aren't public or presigned and need the same credentials as your API call, check Send authentication when downloading files. integrator.io then includes the auth headers from the initial request on each download.
-
integrator.io downloads the files and adds
fileMetaandblobKeydata to the record -- preview the export and you'll see them in the response. Note the exact path where theblobKeylands (it sits with the URL field it came from, so a URL atpresigned_urlyieldspresigned_url.blobKey), because the destination's Blob key path has to match it. Send the files onward with use case 1 (file providers) or use case 4 (HTTP).
Docs: Export data from an HTTP source application and a full example flow.
Use case 3: the API returns file content in the response (often base64)
Gmail attachments, NetSuite RESTlets, and label-generation APIs do this: the file arrives as raw bytes or as a base64 string inside a JSON field. Don't fetch it with a record operation -- you'll hit the 5 MB record cap and end up holding a string no transfer step can use.
-
Start the flow with a record export (the list of messages or documents -- an HTTP flow can't start with a transfer).
-
Add a Look up additional files (per record) step and configure the same API call you'd have made anyway, using values from the record in the relative URI.
-
If the response body is the raw file, you're done with the fetch. If the file content sits inside a JSON field (Gmail's
data, for example), open Non-standard API response patterns and set Path to file in HTTP response body to that field. -
If the content is encoded, set File encoding -- base64 in almost every case.
-
Add a response mapping on the lookup: extract
data[0].blobKey, generateblobKey. Without this, theblobKeynever lands on your record and the downstream transfer has nothing to reference. -
Send the file onward with use case 1 or use case 4.
Here's the Gmail version, since it comes up constantly. The lookup calls GET /gmail/v1/users/me/messages/{{record.messageId}}/attachments/{{record.attachmentId}}, and Gmail answers with the file as a base64 string inside JSON:
{
"size": 48211,
"data": "JVBERi0xLjcKJeLjz9MKNCAwIG9iago8PC9GaWx0ZXIvRmxhdGVEZWNvZG..."
}
On the file lookup, set Path to file in HTTP response body to data and File encoding to base64, then response-map the blobKey. The attachment is now a real file in your flow.
This lookup is also the standard converter whenever you're holding base64 and need a blobKey: point a file lookup at any endpoint that returns the content, set the path and encoding, and the platform stores the file for you.
Use case 4: send a file to an HTTP API
The destination step is Transfer files into destination application on your HTTP connection. Two things are always true: a prior step must have produced a blobKey (use cases 1-3), and Blob key path under Advanced settings must point at it. What varies is the body format the API expects.
A word on Blob key path syntax, because it's a frequent source of silent failures:
-
The path is relative to the record root. If the key sits at the top level, it's
blobKey. If you response-mapped it deeper, use dot notation:nested.blobKey. -
Don't prefix with
record.the way you would in a handlebars expression --record.blobKeyresolves to nothing. -
JSONPath isn't supported here.
$.nested.blobKeyalso resolves to nothing. -
Worst of all, a path that resolves to nothing doesn't produce an error. We tested this: the run reports success and the destination receives a blank file. If your files are arriving empty, check this field first.
Raw file as the body. The default. The file bytes stream as the request body -- nothing else to configure beyond method and URI.
multipart/form-data. Set the request media type to Multipart / form-data, then define the body as a JSON array of parts:
[
{
"name": "file",
"value": "{{blob}}",
"type": "attachment",
"filename": "{{fileMeta.fileName}}"
},
{
"name": "documentType",
"value": "invoice",
"type": "inline"
}
]
-
The file part must be
"type": "attachment"with"value": "{{blob}}"-- that placeholder is where integrator.io streams the file bytes from yourblobKey, and it's the only value the platform accepts for an attachment. -
Regular form fields ride along as
"type": "inline"parts. An inline part whose value is a JSON object needs{{{jsonEncode (jsonSerialize field)}}}. -
Don't write a MIME boundary anywhere. The platform assembles the multipart body and generates the boundary itself.
Docs: Upload a file or record as multipart/form-data and Import files into an HTTP destination app.
Anything else (the escape hatch). Some APIs want a multipart flavor the form-data builder doesn't produce -- Google Drive's multipart/related upload is the classic, and Gmail's send-message upload works the same way. For those, take full manual control:
-
Set Override request media type to Plain text, so the platform stops formatting the body for you.
-
Add a
Content-Typeheader carrying the real media type and a boundary you invent:multipart/related; boundary=my-file-boundary. -
Write the entire MIME document as the HTTP request body, with
{{blob}}where the file bytes go. Keep Blob key path set -- that's still where the bytes come from.
Here's the body for a Google Drive upload (POST /upload/drive/v3/files?uploadType=multipart):
--my-file-boundary
Content-Type: application/json; charset=UTF-8
{
"name": "invoice-1042.pdf",
"parents": ["1AbCdEfGhIjKlMnOp"]
}
--my-file-boundary
Content-Type: application/pdf
{{blob}}
--my-file-boundary--
Every boundary line in the body must match the boundary declared in the header -- mind the leading -- on each one and the extra trailing -- on the last. Handlebars still evaluate here, so the metadata can be dynamic ("name": "{{fileMeta.fileName}}"), and {{blob}} still streams the real bytes at send time. For Google Drive specifically, the built-in folder transfer from use case 1 is the first choice -- reach for the raw API call only when you need request options the folder mode doesn't expose.
Use case 5: turn records into a file, then send it to an HTTP API
File-provider imports can generate a file from the records in your flow (CSV, JSON, XML, XLSX). The generic HTTP import can't do that aggregation today -- so when the destination is an HTTP API, stage the file and split the work across two flows:
-
Flow 1 exports your records and imports them to Celigo Storage through the Celigo APIs connector, generating the file. Celigo Storage is the platform-managed file store under Resources > File storage -- no external SFTP server or S3 bucket to stand up, though any file provider you already have works for staging too.
-
Flow 2 picks the staged file up from Celigo Storage with Transfer files out of source application -- and does not parse it. Parsing would turn the file back into the records you started with; leaving it unparsed hands you a
blobKey, which is what use case 4 needs to stream the finished file to the destination.
This staging pattern is also the general answer whenever a file has to cross flows: blobKeys are tied to the run that created them, so the file itself has to land somewhere durable in between.
Use case 6: the API wants base64 inside a JSON body
Some APIs won't take raw file bytes at all -- they want a JSON payload with the file embedded as a base64 string (AWS Textract's Document.Bytes is a classic). Counterintuitively, this is still a transfer job, not a record import: {{blob}} only exists on transfer steps, so a record import has no way to reach the file's content.
-
Get a
blobKeyonto the record first (use cases 1-3). -
Destination: Transfer files into destination application, with Blob key path set as usual.
-
Hand-write the JSON payload in the HTTP request body, with
{{blob}}in the string spot where the base64 belongs:
{
"fileName": "{{fileMeta.fileName}}",
"content": "{{blob}}"
}
- Set Character encoding to
base64, so the platform encodes the file bytes before dropping them in where{{blob}}sits.
The record itself stays small -- the file rides through {{blob}}, so the 5 MB record cap isn't the constraint here; the destination API's own payload limit is.
One exception: if a base64 string is already sitting in a record (it arrived that way and never became a blobKey), a plain record import that maps the string into the body works too -- but then the 5 MB record cap governs the record carrying it, so convert to a blobKey (use case 3) for anything sizable.
The absolute last resort: converting between blobKey and base64 on the record. If none of the patterns above fit and you truly need the file's base64 as a record field -- for mapper logic, a script, or a payload no transfer body can express -- you can bounce the file off integrator.io's mirror API. A Transfer files into destination application step pointed at POST /v1/mirror with body {"blobBase64": "{{blob}}"} and Character encoding set to base64 gets the payload echoed straight back; a response mapping (_json.blobBase64) then lands the string on the record. The reverse direction works the same way with use case 3's lookup pattern pointed at the mirror. Both directions put the file's content inside a record, so the 5 MB cap applies in full -- if you find yourself here, first double-check that one of the earlier use cases doesn't already cover you.
Errors, decoded
| Error | What it means | Fix |
|---|---|---|
Could not locate the blob object with blobKey |
The path found a value, but it isn't a live key -- usually a blobKey from a different flow run, which doesn't carry over |
Produce the blob in the same flow, or stage the file (use case 5) |
Record or response exceeded 5 MB (response stream exceeded limit...) |
File bytes are traveling through the record lane | Fetch as a file instead (use cases 2 and 3) |
Invalid value for "value" field. If type="attachment" then accepted values... "{{blob}}" |
A multipart attachment part has something other than {{blob}} as its value |
Set the file part to "type": "attachment", "value": "{{blob}}" (use case 4) |
Destination received the literal text blobKey or a tiny corrupt file |
The body references the pointer as data instead of letting the platform stream the file | Use a transfer step with Blob key path set, and {{blob}} in multipart bodies |
| Run reports success but the destination file is blank | Blob key path didn't resolve -- a record. prefix, JSONPath, or a typo. Unresolvable paths fail silently |
Use plain dot notation relative to the record root: blobKey, or nested.blobKey for nested keys |
| Transfer step fails and you only have a base64 string | Transfer steps require a blobKey; a base64 string isn't one |
Convert it with a file lookup (use case 3); if it must stay a string, map it in a record import (use case 6) |
Worked examples from this forum
-
NetSuite file cabinet to SharePoint -- includes an installable flow ZIP
If your scenario doesn't fit any of these, reply with what your API returns (or expects) and we'll figure out which lane it belongs in.