Integrations

Integrations are the core building blocks of Retrieve. Each integration represents a workflow that moves and transforms data between systems.

What is an Integration?

An integration is a configured workflow that:

  • Connects systems: Links your business applications together
  • Moves data: Transfers information between platforms
  • Transforms data: Converts data formats and structures
  • Runs automatically: Executes on schedules or triggers
  • Handles errors: Manages failures and retries gracefully

Integration Components

1. Workflow Nodes

Each integration consists of one or more nodes in a workflow. Each node:

  • Uses an integration package (e.g., Shopify, Odoo, Salesforce)
  • Executes a specific action (pull or push data)
  • Has its own credentials and configuration
  • Can apply field mapping and transformations
  • Passes data to the next node via Redis

2. Credentials

Each node requires authentication credentials for its platform. You have two options:

Node-Specific Credentials

Credentials scoped to a specific node, useful when you need different credentials for different operations:

{
  "node-1": {
    "shopify": {
      "api_secret_key": "your_api_secret_key",
      "admin_api_access_token": "your_admin_api_access_token",
      "host_name": "your-store.myshopify.com"
    }
  }
}

General Credentials

Credentials available to all nodes using that package:

{
  "shopify": {
    "api_secret_key": "your_api_secret_key",
    "admin_api_access_token": "your_admin_api_access_token",
    "host_name": "your-store.myshopify.com"
  }
}

3. Configuration

Each node has configuration options specific to its integration package:

  • API endpoints: Base URLs and API versions
  • Rate limits: Throttling and batch sizes
  • Filters: Which records to process
  • Options: Package-specific settings

4. Scheduling

Control when your integration runs:

  • Every X minutes: Real-time sync (e.g., every 5, 15, 30 minutes)
  • Hourly: Run at a specific minute past the hour
  • Daily: Execute at a specific time each day
  • Weekly: Run on specific days at specific times
  • Cron Expression: Advanced scheduling with full cron syntax
  • Webhook Trigger: Run when an external event occurs
  • Manual: Execute on-demand only

Integration Lifecycle

1. Design Phase

Plan your integration:

  1. Identify source and destination systems
  2. Determine what data needs to move
  3. Define field mappings and transformations
  4. Plan error handling and validation

2. Configuration Phase

Set up your integration:

  1. Create a new integration in the dashboard
  2. Add workflow nodes for each step
  3. Configure credentials for each system
  4. Set up field mappings via the UI
  5. Add custom logic with JavaScript functions
  6. Configure scheduling

3. Testing Phase

Validate your integration:

  1. Run a test execution with sample data
  2. Review the output at each node
  3. Check field mappings are correct
  4. Verify data arrives in the destination
  5. Test error scenarios

4. Activation Phase

Deploy your integration:

  1. Review all configurations
  2. Enable the integration
  3. Monitor the first few executions
  4. Verify scheduled runs work as expected

5. Monitoring Phase

Keep your integration healthy:

  • Review execution logs regularly
  • Monitor success/failure rates
  • Watch for API errors or rate limits
  • Track data volume and processing time
  • Set up alerts for failures

Integration Types

Simple Integration (2 Nodes)

Direct data transfer between two systems:

Example: Sync new Shopify customers to Mailchimp subscribers

Multi-Step Integration (3+ Nodes)

Complex workflow with transformation and enrichment:

Example: Pull orders from Shopify → Validate inventory in ERP → Create shipment in logistics system

Multi-Destination Integration

Distribute data to multiple systems:

Example: Pull products from master catalog → Push to Shopify, Amazon, and eBay simultaneously

Best Practices

Naming Conventions

  • Use descriptive names: "Shopify-to-NetSuite-Orders" instead of "Integration-1"
  • Include direction: "Pull-Customers-from-Shopify"
  • Specify data type: "Product-Sync" or "Order-Processing"
  • Use consistent naming across related integrations

Error Handling

  • Always validate data before pushing to destinations
  • Use rewrite functions to filter out invalid records
  • Set up notifications for failed integrations
  • Configure retry logic for temporary failures
  • Log detailed error messages for debugging

Performance

  • Use appropriate scheduling intervals (don't over-sync)
  • Implement pagination for large data sets
  • Filter data at the source to reduce processing
  • Use webhooks for real-time updates instead of polling
  • Monitor API rate limits and adjust batch sizes

Security

  • Store credentials securely using the platform's credential system
  • Use node-specific credentials when connecting to multiple accounts
  • Regularly rotate API keys and tokens
  • Review integration logs for suspicious activity
  • Limit field mappings to only necessary data

Maintenance

  • Document your integration's purpose and data flow
  • Test integrations after API updates from vendors
  • Keep track of API version changes
  • Archive or disable unused integrations
  • Review and update field mappings as systems evolve

Common Integration Patterns

Data Synchronization

Keep data in sync between two systems:

  • One-way sync: Data flows in one direction only
  • Two-way sync: Data flows in both directions (requires careful conflict handling)
  • Incremental sync: Only sync changed records
  • Full sync: Re-sync all records periodically

Data Migration

One-time transfer of data from old to new system:

  • Run once or on-demand
  • Include data validation and cleanup
  • Map old schema to new schema
  • Verify data integrity after migration

Data Enrichment

Enhance data with information from multiple sources:

  • Pull base data from one system
  • Fetch additional details from other systems
  • Combine and transform data
  • Push enriched data to destination

Event-Driven Integration

React to events in real-time:

  • Receive webhooks from source system
  • Process event data immediately
  • Trigger actions in destination systems
  • Send notifications or alerts

Next Steps