Rewrite Functions

Rewrite functions are the most powerful feature in Retrieve. They don't just transform data - they replace the entire action code, giving you complete control over how an integration node operates.

🚀 Complete Control

When you add a rewrite function to a node, your code runs instead of the default action. You can:

Customize API calls with your own parameters and logic
Change how data is fetched or sent
Add complex business logic and validations
Make additional API calls to other services
Implement retry logic, error handling, or custom workflows
Do literally anything the integration package allows

What Happens Without a Rewrite Function?

By default, each action (like "pull_orders" or "push_customers") runs pre-built code from the integration package. This code:

Makes standard API calls with default parameters
Fetches or sends data in a predefined way
Returns data in a standard format

This works great for 90% of use cases. But what if you need custom filtering, special API parameters, or unique logic?

What Happens With a Rewrite Function?

When you add a rewrite function:

❌ The default action code does not run
✅ Your rewrite function runs instead
✅ You have access to all the same tools (API clients, helpers, configuration)
✅ You decide exactly what happens in that node
✅ The workflow continues normally after your function completes

Using Templates to Get Started

Every integration package provides templates - pre-built rewrite functions that show you the base code for each action. Templates are your starting point for customization.

Where to Find Templates

Templates are located in each package's templates/workflow/ directory:

# Magento 2 Package
/packages/magento2/templates/workflow/
  ├── pull_orders.js
  ├── pull_customers.js
  ├── push_orders.js
  ├── push_customers.js
  ├── create_invoice.js
  └── cancel_order.js

# Other packages follow the same pattern
/packages/shopify/templates/workflow/
/packages/netsuite/templates/workflow/

Common Template Files

📥 Pull Action Templates

pull_orders.js
pull_customers.js
pull_products.js

📤 Push Action Templates

push_orders.js
push_customers.js
push_products.js
create_invoice.js
cancel_order.js
create_shipment.js

Note: Creating invoices, orders, shipments, etc. are all push actions since you're sending data to perform an operation.

How to Use a Template

Copy the template for the action you want to customize
Paste it into your node's rewrite function field
Modify the parts you need to customize
Test your changes

Real Template Example: Pull Orders

Here's what a real template looks like. This is the base code for pulling orders from Magento 2:

/**
 * WORKFLOW MODE REWRITE TEMPLATE
 * 
 * This template is for workflow mode integrations (rewrite_function).
 * In workflow mode, the WorkflowOrchestrator automatically:
 * - Executes after_function if defined on the node
 * - Applies field_mapping transformations if defined on the node  
 * - Queues next workflow node(s) automatically
 */

/* Available variables */
let job = jobData.job;
let integration = jobData.integration;
let integrationHelper = jobData.integrationHelper;
let ValidatorHelper = jobData.ValidatorHelper;
let queueManager = jobData.queueManager;
let workflowOrchestrator = jobData.workflowOrchestrator;
let currentNode = jobData.currentNode;
let Moment = jobData.Moment;
let MagentoApi = jobData.MagentoApi;

/* Get the Magento 2 integration details */
const nodeId = currentNode?.id || null;
const magentoIntegration = integrationHelper.getNodeConfig(integration, nodeId, 'magento2');

/* Validate the API credentials */
let validatorHelper = new ValidatorHelper();

/* Initialize the magento client */
let magentoClient = new MagentoApi(validatorHelper.buildConfig(magentoIntegration));

let currentDate = Moment().format('YYYY-MM-DD H:mm:ss');

/**
 * Parameters used to filter the api call
 * ⬇️ THIS IS WHAT YOU'D CUSTOMIZE ⬇️
 */
let params = {
    "filter_groups": [
        {
            "filters": [
                {
                    "field": "created_at",
                    "value": magentoIntegration['orders_last_pull'],
                    "condition_type": "gteq"
                }
            ]
        }
    ],
    "sortOrders": [
        {
            "field": "created_at",
            "direction": "asc"
        }
    ]
};

/**
 * Fetch orders from Magento
 */
let response = await magentoClient.get('orders', params);

// Validate response
if (!response.data.total_count || !response.data.items) {
    return {
        jobStatus: 0,
        message: "API call failed",
        error: JSON.stringify(response.data)
    };
}

// Return the data
return {
    jobStatus: 1,
    data: response.data.items,
    qty: response.data.total_count
};

Key Points:

Line 1-15: Comments explaining what the template does
Line 17-35: Available variables (job, integration, helpers, API client)
Line 37-53: Configuration and validation setup
Line 55-71: API call parameters (this is what you'd customize!)
Line 73-75: The actual API call
Line 77-82: Response validation

Available Variables in Rewrite Functions

Every rewrite function receives a jobData object with powerful tools:

`job`

The current job object containing data from the previous node

// Access data from previous node
let previousData = job.data.data;

// Access job metadata
let jobId = job.id;

`integration`

The entire integration configuration

// Access integration config
let config = integration.config;

// Access credentials
let credentials = integration.credentials;

`integrationHelper`

Helper functions for common operations

// Get node-specific config
let nodeConfig = integrationHelper.getNodeConfig(integration, nodeId, 'magento2');

// Format dates, strings, etc.
let formatted = integrationHelper.formatDate(date);

`currentNode`

Information about the current workflow node

// Get current node info
let nodeId = currentNode.id;
let nodeName = currentNode.name;
let nodeAction = currentNode.action;

`queueManager`

For advanced workflow control

// Add custom jobs to queue
queueManager.addToQueue(jobData);

`workflowOrchestrator`

Manages workflow execution flow

// Access workflow control
workflowOrchestrator.processNextNode(data);

Package-Specific Variables

Each integration package provides its own API client and helpers:

// Magento 2 specific
let MagentoApi = jobData.MagentoApi;
let magentoClient = new MagentoApi(config);

// Shopify specific  
let ShopifyApi = jobData.ShopifyApi;
let shopifyClient = new ShopifyApi(config);

// Each package provides its own API client

Third-Party Libraries

Common libraries are pre-loaded:

// Moment.js for date handling
let Moment = jobData.Moment;
let date = Moment().format('YYYY-MM-DD');

// JSON-cycle for circular references
let jc = jobData.jc;

Common Customization Scenarios

Scenario 1: Custom API Filters

Problem: You only want orders from a specific store or with a specific status

Solution: Modify the API parameters in the template

❌ Default Template

// Default: Pull all orders since last run
let params = {
    "filter_groups": [
        {
            "filters": [
                {
                    "field": "created_at",
                    "value": lastPullDate,
                    "condition_type": "gteq"
                }
            ]
        }
    ]
};

✅ Your Customization

// Custom: Only orders from store 2 with status "processing"
let params = {
    "filter_groups": [
        {
            "filters": [
                {
                    "field": "created_at",
                    "value": lastPullDate,
                    "condition_type": "gteq"
                },
                {
                    "field": "store_id",
                    "value": "2",
                    "condition_type": "eq"
                },
                {
                    "field": "status",
                    "value": "processing",
                    "condition_type": "eq"
                }
            ]
        }
    ]
};

Scenario 2: Data Transformation Before Sending

Problem: Need to format data differently before pushing to destination

Solution: Add transformation logic in your rewrite function

// Push customers template with custom transformation
let customers = job.data.data;
let results = {};

for (let key in customers) {
    let customer = customers[key];
    
    // Custom transformation before sending
    let transformedCustomer = {
        customer: {
            id: customer.id,
            email: customer.email,
            firstname: customer.first_name?.toUpperCase(), // Uppercase
            lastname: customer.last_name?.toUpperCase(),
            // Add custom attribute
            custom_attributes: [
                {
                    attribute_code: "source",
                    value: "integration_import"
                },
                {
                    attribute_code: "import_date",
                    value: new Date().toISOString()
                }
            ]
        }
    };
    
    try {
        let response = await magentoClient.put(`customers/${customer.id}`, transformedCustomer);
        results[key] = { result: true, item: response.data };
    } catch (e) {
        results[key] = { result: false, error: e.toString() };
    }
}

return {
    jobStatus: 1,
    items: results
};

Scenario 3: Conditional Logic

Problem: Different handling based on data conditions

Solution: Add if/else logic in your rewrite function

// Pull orders with different handling based on order value
let response = await magentoClient.get('orders', params);
let orders = response.data.items;

let processedOrders = orders.map(order => {
    let total = parseFloat(order.grand_total);
    
    // Add priority based on order value
    if (total > 1000) {
        order.priority = 'high';
        order.requires_approval = true;
        order.notification_email = 'manager@company.com';
    } else if (total > 500) {
        order.priority = 'medium';
        order.requires_approval = false;
    } else {
        order.priority = 'low';
        order.requires_approval = false;
    }
    
    // Add calculated shipping
    order.shipping_category = total > 100 ? 'free' : 'standard';
    
    return order;
});

return {
    jobStatus: 1,
    data: processedOrders,
    message: `Processed ${processedOrders.length} orders`
};

Scenario 4: External API Enrichment

Problem: Need to fetch additional data from another service

Solution: Make additional API calls in your rewrite function

const axios = require('axios');

// Pull orders and enrich with external data
let response = await magentoClient.get('orders', params);
let orders = response.data.items;

for (let order of orders) {
    try {
        // Fetch shipping rate from external API
        let shippingResponse = await axios.post('https://api.shipping.com/rates', {
            weight: order.weight,
            zip: order.shipping_address?.postcode,
            country: order.shipping_address?.country_id
        }, {
            headers: { 'Authorization': 'Bearer YOUR_API_KEY' }
        });
        
        order.shipping_rate = shippingResponse.data.rate;
        order.shipping_carrier = shippingResponse.data.carrier;
        order.estimated_delivery = shippingResponse.data.estimated_days;
        
    } catch (error) {
        // Fallback if shipping API fails
        order.shipping_rate = 10.00;
        order.shipping_carrier = 'standard';
        order.shipping_api_error = error.message;
    }
}

return {
    jobStatus: 1,
    data: orders
};

Scenario 5: Batch Processing

Problem: Need to process items in smaller batches for API limits

Solution: Add batching logic

// Push products in batches of 50 to respect API limits
let products = job.data.data;
let batchSize = 50;
let results = {};

// Split into batches
for (let i = 0; i < products.length; i += batchSize) {
    let batch = products.slice(i, i + batchSize);
    
    console.log(`Processing batch ${i / batchSize + 1} of ${Math.ceil(products.length / batchSize)}`);
    
    for (let key in batch) {
        let product = batch[key];
        
        try {
            let response = await magentoClient.post('products', { product });
            results[i + parseInt(key)] = {
                result: true,
                sku: product.sku
            };
        } catch (e) {
            results[i + parseInt(key)] = {
                result: false,
                sku: product.sku,
                error: e.toString()
            };
        }
    }
    
    // Small delay between batches to avoid rate limits
    await new Promise(resolve => setTimeout(resolve, 1000));
}

return {
    jobStatus: 1,
    items: results,
    message: `Processed ${products.length} products in ${Math.ceil(products.length / batchSize)} batches`
};

Return Format Requirements

⚠️ Critical: Always Return Arrays

All actions must return data as an array of records, never a single object. This ensures consistent data handling throughout the workflow.

// ✅ CORRECT - Data is an array
return {
  jobStatus: 1,
  data: [
    { id: 1, name: 'Product A', price: 29.99 },
    { id: 2, name: 'Product B', price: 39.99 },
    { id: 3, name: 'Product C', price: 49.99 }
  ]
}

// ✅ CORRECT - Even for single record
return {
  jobStatus: 1,
  data: [
    { id: 1, name: 'Single Product', price: 29.99 }
  ]
}

// ✅ CORRECT - Empty result
return {
  jobStatus: 1,
  data: []
}

// ❌ INCORRECT - Data is not an array
return {
  jobStatus: 1,
  data: { id: 1, name: 'Product', price: 29.99 }
}

Job Status Codes

jobStatus: 1

Success - Workflow continues to next node

jobStatus: 0

Error - Workflow stops execution

jobStatus: 2

Warning - Workflow continues but logs issue

Return Object Structure

return {
  jobStatus: 1,              // Required: 1 = success, 0 = error, 2 = warning
  data: [...],               // For pull actions: array of items
  items: {...},              // For push actions: results object
  message: "Optional message",
  qty: 10,                   // Optional: total count
  errors: [],                // Optional: array of errors
  metadata: {}               // Optional: additional info
};

How Rewrite Functions Work with Other Features

Rewrite Function + Field Mapping

Your rewrite function runs first, then field mapping is applied to the results:

// Your rewrite function returns this:
return {
  jobStatus: 1,
  data: [
    { shopify_id: 123, shopify_email: "john@example.com" }
  ]
};

// Then field mapping transforms it to:
{
  customer_id: 123,           // Mapped from shopify_id
  email: "john@example.com"   // Mapped from shopify_email
}

Rewrite Function + After Origin Function

Both can be used together for maximum control:

Rewrite function replaces the default action (fetches/sends data your way)
After Origin function then transforms the results
Field mapping applies any final field-level transformations
Data moves to the next node

Best Practices

1. Start with a Template

✅ Always copy the template for your action as a starting point
✅ Templates include proper error handling and API client setup
❌ Don't write from scratch unless absolutely necessary

2. Modify Only What You Need

✅ Change API parameters, add filters, customize logic
✅ Keep the core structure (variables, API client, return format)
❌ Don't remove error handling or validation code

3. Always Return Proper Format

✅ Return {jobStatus: 1, data: [...]} for pull actions
✅ Return {jobStatus: 1, items: {...}} for push actions
✅ Data must be an array, even for single records
❌ Never return just the data without jobStatus

4. Handle Errors Gracefully

✅ Use try-catch blocks for API calls
✅ Return meaningful error messages
✅ Use jobStatus: 0 for critical errors, 2 for warnings

try {
  let response = await magentoClient.get('orders', params);
  
  if (!response.data || !response.data.items) {
    return {
      jobStatus: 2,  // Warning - continue but log
      data: [],
      message: "No orders found",
      warning: "Empty response from API"
    };
  }
  
  return {
    jobStatus: 1,
    data: response.data.items
  };
  
} catch (error) {
  return {
    jobStatus: 0,  // Error - stop workflow
    data: [],
    message: "Failed to fetch orders",
    error: error.message
  };
}

5. Test Thoroughly

✅ Test with real data from your systems
✅ Test edge cases (empty results, errors, large datasets)
✅ Check the workflow continues properly
✅ Verify field mapping still works after rewrite

6. Document Your Changes

✅ Add comments explaining why you customized the code
✅ Document any non-standard API parameters
✅ Note any business logic specific to your use case

// Business requirement: Only pull VIP customer orders (ticket #4567)
// VIP customers have customer_group_id = 4 in Magento
let params = {
    "filter_groups": [
        {
            "filters": [
                {
                    "field": "customer_group_id",
                    "value": "4",
                    "condition_type": "eq"
                }
            ]
        }
    ]
};

7. Keep It Maintainable

✅ Use clear variable names
✅ Break complex logic into smaller functions
✅ Avoid deeply nested conditions
✅ Follow JavaScript best practices

Common Pitfalls to Avoid

❌ Returning Object Instead of Array

// ❌ Wrong - returning single object
return {
  jobStatus: 1,
  data: { id: 1, name: "Product" }
};

// ✅ Correct - wrap in array
return {
  jobStatus: 1,
  data: [{ id: 1, name: "Product" }]
};

❌ Not Handling Null Values

// ❌ Wrong - will crash if address is null
let city = order.shipping_address.city;

// ✅ Correct - safe access
let city = order.shipping_address?.city || "Unknown";

❌ Missing Error Handling

// ❌ Wrong - no error handling
let response = await magentoClient.get('orders', params);
return { jobStatus: 1, data: response.data.items };

// ✅ Correct - wrapped in try-catch
try {
  let response = await magentoClient.get('orders', params);
  return { jobStatus: 1, data: response.data.items };
} catch (error) {
  return { jobStatus: 0, message: error.message };
}

❌ Forgetting Async/Await

// ❌ Wrong - missing await
let response = magentoClient.get('orders', params);

// ✅ Correct - use await
let response = await magentoClient.get('orders', params);

Real-World Complete Example

Scenario: Pull orders from Magento but only for VIP customers, add calculated shipping costs, and filter out cancelled orders

/**
 * Custom Order Pull for VIP Customers
 * - Only pulls orders from VIP customer group (group_id = 4)
 * - Filters out cancelled orders
 * - Calculates shipping costs based on weight
 * - Adds priority flags for high-value orders
 */

let job = jobData.job;
let integration = jobData.integration;
let integrationHelper = jobData.integrationHelper;
let ValidatorHelper = jobData.ValidatorHelper;
let currentNode = jobData.currentNode;
let Moment = jobData.Moment;
let MagentoApi = jobData.MagentoApi;

// Setup
const nodeId = currentNode?.id || null;
const magentoIntegration = integrationHelper.getNodeConfig(integration, nodeId, 'magento2');
let validatorHelper = new ValidatorHelper();
let magentoClient = new MagentoApi(validatorHelper.buildConfig(magentoIntegration));

// Custom parameters: VIP customers only, not cancelled
let params = {
    "filter_groups": [
        {
            "filters": [
                {
                    "field": "created_at",
                    "value": magentoIntegration['orders_last_pull'],
                    "condition_type": "gteq"
                },
                {
                    "field": "customer_group_id",
                    "value": "4",  // VIP group
                    "condition_type": "eq"
                },
                {
                    "field": "status",
                    "value": "canceled",
                    "condition_type": "neq"  // Exclude cancelled
                }
            ]
        }
    ]
};

try {
    // Fetch orders
    let response = await magentoClient.get('orders', params);
    
    if (!response.data || !response.data.items) {
        return {
            jobStatus: 1,
            data: [],
            message: "No VIP orders found"
        };
    }
    
    let orders = response.data.items;
    
    // Process each order with custom logic
    let processedOrders = orders.map(order => {
        let total = parseFloat(order.grand_total);
        let weight = parseFloat(order.weight || 0);
        
        // Calculate shipping cost based on weight
        let shippingCost = 0;
        if (weight <= 5) {
            shippingCost = 10;
        } else if (weight <= 20) {
            shippingCost = 25;
        } else {
            shippingCost = 50;
        }
        
        // Add priority for high-value orders
        order.vip_customer = true;
        order.calculated_shipping = shippingCost;
        order.priority = total > 500 ? 'urgent' : 'normal';
        order.requires_manager_approval = total > 1000;
        
        return order;
    });
    
    return {
        jobStatus: 1,
        data: processedOrders,
        qty: processedOrders.length,
        message: `Successfully pulled ${processedOrders.length} VIP orders`
    };
    
} catch (error) {
    return {
        jobStatus: 0,
        data: [],
        message: "Failed to pull VIP orders",
        error: error.message
    };
}

Getting Help

If you're stuck customizing a rewrite function:

📁 Check the templates/workflow/ directory for your package
📖 Review similar templates for patterns and examples
🔍 Look at the package's main queue processor files for API methods
💬 Contact support with your specific use case
📝 Share your rewrite function code for debugging help

Next Steps

Field Mapping - Combine rewrite functions with field mapping
Understanding Workflows - Use rewrite functions in multi-node workflows
Actions (Pull/Push) - Understand what actions you can rewrite
Integration Packages - Find templates for specific packages