I am trying to list all contacts from HubSpot, and their API endpoint has a rate limit. I get the error 'TEN_SECONDLY_ROLLING' which means I need to limit the number of requests.
Is there a way to add a delay between the PAGE requests that Celigo makes? I did not find relevant info here: https://docs.celigo.com/hc/en-us/articles/360043927292-Fine-tune-integrator-io-for-optimal-data-throughput
@vreddhibhat we just rolled out functionality for automatically recovering from rate limit errors, but as you see it currently doesn't work on exports, only imports. In some upcoming release that should be enhanced so you won't have to mess with this issue.
That being said, to currently do this, I think your best option would be to create a custom connection to HubSpot, using the universal HTTP connector. Under the connection settings, there is an option to specify a time between api calls. Here you could add some delay between requests to avoid the rate limit.
Hopefully early next year, all connectors will be migrated over to the newer HTTP framework so you wouldn't need to make a custom connection. Significant progress has already been made in that migration, but HubSpot hasn't yet made it.
Tyler- A followup question on your response. I am also receiving the rate limit error on an export from HubSpot, but it appears that all errors are being autoresolved and the steps are completing successfully. Can you help make sense of if indeed the data "flow" is successful, or if any remediation is needed due to the errors.
Screenshot showing the rate limit error (note these are falling into the "Resolved errors" bucket and are auto-resolved.
Screenshot showing the flows successful on each step
@daveguderian in your case, the rate limiting is happening on the lookup itself which does support auto resolve of rate limiting errors when one-to-many is not configured. So we would have received the initial rate error, then backed off our concurrent requests, then pinged the endpoint again to get the needed data. It got auto resolved because we were able to successfully handle the initial rate error and get the needed data in a subsequent call.
That being said, it looks like you don't really even need this lookup step because you can get all properties for a company in your initial HubSpot call. You just have to specify the fields you want by putting them in a comma separated list in a parameter field.
Default without specifying properties:
If specifying properties:
If you want to get a list of all properties for an object, you can use the following endpoint (or look in your HubSpot account):
https://developers.hubspot.com/docs/api/crm/properties
crm/v3/properties/{objectType}
Hey Tyler-
Thats a great idea, but unfortunately I think we have too many properties (we have 500 for our contacts object, and almost 200 for companies) and may be hitting the limit. Below is the error I receive when I try to put them into the "Configure Search Parameters" field.
@daveguderian looks like an internal text field size limit. However, you can work around it by using settings:
/crm/v3/objects/companies?properties={{{settings.export.properties}}}{
"properties": "hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore"
}
@daveguderian if you want to get really crazy, you could create a custom form that would allow you to multi select the fields you want (I know in your case you just want to select all).
import {
connections,
integrations,
exports,
imports,
flows,
request
} from 'integrator-api'
class Form {
// { fieldMap: {}, layout: { fields: [] } }
constructor() {
this.fieldMap = {};
this.layout = {};
this.layout.containers = [];
}
fields(...fields) {
this.layout.fields = fields;
}
container(type, label, ...fields) {
this.layout.containers.push({
type: type,
containers: [{
label: label,
fields: fields
}]
});
}
column(label, ...fields) {
this.layout.type = 'column';
this.layout.containers.push({
label: label,
fields: fields
});
}
}
class Field {
constructor(id, name, label, type) {
this.id = id;
this.name = name;
this.label = label;
this.type = type;
}
isRequired() {
this.required = true;
}
removeInvalidValues() {
this.removeInvalidValues = true;
}
canBeDeleted() {
this.showDelete = true;
}
isMultiline() {
this.multiline = true;
}
description(description) {
this.description = description;
}
help(helpText) {
this.helpText = helpText;
}
typeOfInput(type) {
this.inputType = type;
}
default(defaultValue) {
this.defaultValue = defaultValue;
}
delimiter(delimiter) {
this.delimiter = delimiter;
}
keyNamePlaceholder(keyName) {
this.keyName = keyName;
}
valueNamePlaceholder(valueName) {
this.valueName = valueName;
}
mode(mode) {
this.mode = mode;
}
addOption(arg1, arg2) {
if (!this.hasOwnProperty('options')) {
this.options = [{
items: []
}];
}
if (arg2) {
this.options[0].items.push({
label: arg1,
value: arg2
});
} else {
this.options[0].items.push(arg1);
}
}
addStaticMapOptions(keyName, keyLabel, keyOptions, valueName, valueLabel, valueOptions) {
this.keyName = keyName;
this.keyLabel = keyLabel;
this.keyOptions = keyOptions;
this.valueName = valueName;
this.valueLabel = valueLabel;
this.valueOptions = valueOptions;
}
}
function formInit(options) {
let form = new Form();
// reference: https://docs.celigo.com/hc/en-us/articles/360059205112-Common-form-fields
// deconstruct form to make access to fieldMap for prop assignment simpler
let { fieldMap } = form;
let selectedObject = '';
let regex = /(?<=objects\/)[^\/?]+/;
let match = options.resource.http.relativeURI.match(regex);
if (match) {
selectedObject = match[0];
}
//create field for staticMap
fieldMap.fieldList = new Field('fieldList','fieldList','Fields(s)', 'multiselect');
if (selectedObject) {
let selectOptions = getOptions(options.resource._connectionId,selectedObject);
for (let s of selectOptions) {
fieldMap.fieldList.addOption(s.label,s.value);
}
}
fieldMap.fieldList.description('Select the fields you want to select.');
fieldMap.fieldList.isRequired();
fieldMap.fieldList.removeInvalidValues();
//set the layout for the form
form.fields(...['fieldList']);
// wrap up by assigning form to options.resource.settingsForm prop; return the form as well
options.resource.settingsForm.form = form;
return form;
}
function getOptions(hubspotConnectionId,selectedObject) {
return exports.runVirtual({
"export": {
"name": "Get all fields",
"_connectionId": hubspotConnectionId,
"asynchronous": true,
"oneToMany": false,
"sandbox": false,
"http": {
"relativeURI": `/crm/v3/properties/${selectedObject}`,
"method": "GET",
"successMediaType": "json",
"errorMediaType": "json",
"formType": "http",
"paging": {
"method": "token",
"path": "paging.next.after",
"relativeURI": `/crm/v3/properties/${selectedObject}?after={{{export.http.paging.token}}}`,
"lastPageStatusCode": 404
},
"response": {
"resourcePath": "results"
}
},
"transform": {
"type": "expression",
"expression": {
"rules": [
[]
],
"rulesTwoDotZero": {
"mappings": [
{
"generate": "label",
"dataType": "string",
"extract": "$.label",
"sourceDataType": "string",
"status": "Active"
},
{
"generate": "value",
"dataType": "string",
"extract": "$.name",
"sourceDataType": "string",
"status": "Active"
}
],
"mode": "create"
},
"version": "2"
},
"rules": [
[]
],
"version": "2"
},
"adaptorType": "HTTPExport"
}
}).data;
}This is awesome Tyler. Like you said I think right now I am just interested in pulling all properties, but I do love this idea for the future and have a couple ideas on how I could use this for some different flows I will be making.
One thing I noticed this morning, was that my "Delta" loads seem to not be working, in the sense that they are pulling all of the data (so I don't think my {{lastExportDateTime}} handlebar is correct. Here is what I have:
/crm/v3/objects/companies?properties={{settings.export.properties}}?{{lastExportDateTime}}Hey @daveguderian, another way of getting Delta data through the Hubspot v3 APIs is by selecting the Search for Companies endpoint and opening the Search Parameters. Once in there, you can add this code and paste it inside the filterGroups section:
[
{
"filters": [
{
"propertyName": "hs_lastmodifieddate",
"operator": "GT",
"value": "{{lastExportDateTime}}"
}
]
}
]Ex:
You can also add your properties here as well. Here is the article from our Help Center that walks through this. Hope that helps!
Hey Kevin - Thanks for the feedback. I am using the list V3 endpoint, and it appears that "filters" is not an option.
@daveguderian If you switch the API endpoint to Search for Companies instead of 'List' you should see the other options
Thanks Kevin. Is there a particular syntax to use if I wanted to use the "List" endpoint?
@daveguderian there is not so you have to use the search option. Use the following setup:
Switch to REST form view. If you don't use the form settings like I showed, just put a string array of property fields you want to use instead of the {{#each}} handlebar block I'm using.
{
"properties": [{{#each settings.export.fieldList}}"{{this}}"{{^if @last}},{{/if}}{{/each}}],
"filterGroups": [
{
"filters": [
{
"propertyName": "hs_lastmodifieddate",
"operator": "GTE",
"value": "{{lastExportDateTime}}"
}
]
}
]
}
{{#if data._PARENT.paging.next.after}}
{
"properties": [{{#each settings.export.fieldList}}"{{this}}"{{^if @last}},{{/if}}{{/each}}],
"filterGroups": [
{
"filters": [
{
"propertyName": "hs_lastmodifieddate",
"operator": "GTE",
"value": "{{lastExportDateTime}}"
}
]
}
],
"after":"{{data._PARENT.paging.next.after}}"
}
{{/if}}
I know this thread is a bit old, but I am now turning these flows on in PROD and trying to do the initial load and running into issues as it appears there is a 10,000 limit. Is there anyway around this (and still be able to grab all properties/data for an object)?
@daveguderian it seems the limit is on the /search endpoint, but not if you use the list endpoint. However, the list endpoint doesn't allow you to put a delta date. So for initial load, you'll need to use the list endpoint and then after initial load swap it to the other endpoint. I would just add 2 exports to your flow, one delta and one not delta. Then schedule just the delta export to run.
https://community.hubspot.com/t5/APIs-Integrations/Error-on-contacts-search-over-10-000-when-using-pagination/m-p/265624/highlight/true#M23745
@daveguderian after office hours today, @tonycurcio and I were brainstorming and came up with a better solution here. The idea is to not use HubSpot's search paging mechanism at all and instead just continuously filter the next pages based on the last record received in the previous api call. We can do this because HubSpot allows us to sort the results based on any field, in this case the lastmodifieddate field. Doing this will allow you to go back to one export for full syncs and delta syncs. Here is the setup below:
{
"properties": [{{#each settings.export.fieldList}}"{{this}}"{{^if @last}},{{/if}}{{/each}}],
"limit":100,
"sorts": [
{
"propertyName": "lastmodifieddate",
"direction": "ASCENDING"
}
],
"filterGroups": [
{
"filters": [
{
"propertyName": "lastmodifieddate",
"operator": "GTE",
"value": "{{lastExportDateTime}}"
}
]
}
]
}
{
"properties": [{{#each settings.export.fieldList}}"{{this}}"{{^if @last}},{{/if}}{{/each}}],
"limit":100,
"sorts": [
{
"propertyName": "lastmodifieddate",
"direction": "ASCENDING"
}
],
"filterGroups": [
{
"filters": [
{
"propertyName": "lastmodifieddate",
"operator": "GT",
"value": "{{{previous_page.last_record.properties.lastmodifieddate}}}"
}
]
}
]
}Settings list of fields (yours may be different):
Tyler and Tony-
Thank you so much for taking the time to think on this. These HubSpot Datalake flows are high maintenance, so I sincerely appreciate you guys looking to make this simpler.
So just to clarify on the above, this one export will replace the step below correct?
If you recall I have two flows for each data element. The first one grabs the properties, and export information from the second flow (see below):
And then the second flow queries HubSpot for the data and places the file in S3. I believe the new step would replace the export step in flow 2, correct?
@daveguderian that's correct
Tyler - What would be the correct steps to modify this step for the full initial load? Would I just need to remove the filtergroups section and change the export type from "Delta" to "All"?
@daveguderian for this one, you can just keep the filter groups and set the datetime when manually running back to 01-01-2000. The reason being is that this api requires paging against a timestamp anyways so if there were ever any null values then it wouldn't pick them up.
