Understanding Workflows

Workflows enable you to build sophisticated, multi-step integrations using a visual node-based interface. This guide explains how workflows work and how to leverage them for complex data integration scenarios.

What are Workflows?

Workflows represent the core integration approach in Retrieve, where data flows through multiple sequential steps, each performing specific operations. Workflows allow you to:

  • Chain Multiple Actions: Execute several operations in sequence
  • Apply Transformations: Modify data at each step with custom logic
  • Route Conditionally: Send data to different paths based on content
  • Process in Parallel: Execute multiple branches simultaneously
  • Enrich Data: Fetch additional information from other systems mid-flow

Core Concepts

Nodes

A node is an individual step in your workflow. Each node:

  • Executes a specific action (pull, push, custom)
  • Can have its own configuration and credentials
  • Applies field mapping to transform data
  • Can execute custom JavaScript code
  • Passes data to the next node(s)

Node Types:

  1. Pull Nodes: Retrieve data from external systems
  2. Push Nodes: Send data to external systems
  3. Transform Nodes: Process and modify data
  4. Decision Nodes: Route data based on conditions
  5. API Call Nodes: Make custom API requests

Data Flow

Data flows through the workflow like this:

  1. Node Execution: Current node executes its action
  2. Data Transformation: Field mapping is applied
  3. Custom Logic: Rewrite function processes data (if defined)
  4. Data Passing: Transformed data is passed to next node(s)
  5. Next Node: Process repeats for connected nodes

Rewrite Functions

Rewrite functions in workflow mode provide complete control over data at each node:

// Available variables:
// - job: Current job information
// - integration: Integration configuration
// - currentNode: Current node configuration
// - queueManager: Queue management helper

// Example: Filter and enrich data
let data = job.data.data;

// Filter out records
let filteredData = data.filter(order => order.status === 'pending');

// Enrich with additional data
for (let order of filteredData) {
  order.processing_date = new Date().toISOString();
  order.priority = order.total > 1000 ? 'high' : 'normal';
}

return {
  jobStatus: 1,
  data: filteredData,
  message: `Processed ${filteredData.length} pending orders`
};

Proceed to Next Nodes

The proceed_to_next_nodes flag controls whether subsequent nodes execute when no data is returned:

  • proceed_to_next_nodes: true (default) - Next nodes always execute, even with empty data
  • proceed_to_next_nodes: false - Next nodes only execute if data is returned

Building a Workflow

Step 1: Plan Your Workflow

Before building, map out:

  1. Data Sources: What systems will you pull from?
  2. Data Flow: How should data move between systems?
  3. Transformations: What changes need to be made?
  4. Destinations: Where does data ultimately go?
  5. Error Handling: What happens if a step fails?

Step 2: Create Workflow Integration

  1. In the dashboard, click "Create New Integration"
  2. Select "Workflow Mode"
  3. Name your integration
  4. Configure general settings

Step 3: Add Nodes

  1. Starting Node: Typically pulls data from a source
  2. Processing Nodes: Transform or validate data
  3. Destination Nodes: Push data to target systems

Real-World Examples

Example 1: Order Fulfillment Pipeline

Goal: Process e-commerce orders through multiple systems

Example 2: Multi-Channel Product Sync

Goal: Distribute products to multiple sales channels

Best Practices

  1. Keep Nodes Focused: Each node should do one thing well
  2. Use Descriptive Names: Name nodes clearly (e.g., "Validate-Inventory")
  3. Handle Empty Data: Always consider what happens with no data
  4. Plan for Errors: Add error handling at each critical step
  5. Document Complex Logic: Comment your rewrite functions

Next Steps