Performance Optimization

Fast integrations mean better user experience, lower costs, and more reliable data syncing. This guide covers best practices and techniques to optimize your Retrieve integrations for maximum performance.

Understanding Integration Performance

Integration performance depends on several factors:

Query Optimization

The most impactful performance optimization is reducing the data you fetch and process.

1. Use Date Filters (Pull Only What Changed)

Instead of pulling all records every time, fetch only records modified since the last sync:

// Fetches ALL orders every time
const orders = await api.get('/orders');
// Returns 10,000 orders (only 50 are new)

// Fetch only orders modified since last sync
const lastSyncTime = await getLastSyncTime();
const orders = await api.get('/orders', {
  params: {
    updated_at_min: lastSyncTime
  }
});
// Returns only 50 new orders

2. Limit Results with Pagination

Process large datasets in smaller batches instead of loading everything at once:

// Process large dataset in batches
let page = 1;
let allOrders = [];

while (true) {
  const response = await api.get('/orders', {
    params: {
      page: page,
      limit: 100 // Fetch 100 at a time
    }
  });
  
  if (response.orders.length === 0) break;
  
  allOrders = allOrders.concat(response.orders);
  page++;
}

return { jobStatus: 1, data: allOrders };

3. Select Only Required Fields

Many APIs allow you to specify which fields to return. Only fetch what you need:

// Returns entire product object with all fields
const products = await api.get('/products');

// Request only needed fields
const products = await api.get('/products', {
  params: {
    fields: 'id,name,price,sku,inventory_quantity'
  }
});

Data Processing Optimization

1. Use Field Mapping Instead of Code

Field mapping is faster than rewrite functions for simple transformations:

// Rewrite function for every record
let orders = job.data.data;

let transformed = orders.map(order => ({
  id: order.id,
  customer: order.customer.name,
  total: order.total_price,
  status: order.status
}));

return { jobStatus: 1, data: transformed };

{
  "id": "id",
  "customer": "customer.name",
  "total": "total_price",
  "status": "status"
}

2. Optimize Loops and Iterations

Efficient code makes a big difference when processing thousands of records:

let orders = job.data.data;
let customers = await getCustomers(); // 1000 customers

// O(n²) - nested loop
let enriched = orders.map(order => {
  let customer = customers.find(c => c.id === order.customer_id);
  return {
    ...order,
    customer_name: customer?.name
  };
});

let orders = job.data.data;
let customers = await getCustomers();

// Build hash map once - O(n)
let customerMap = {};
customers.forEach(c => {
  customerMap[c.id] = c;
});

// Single loop lookup - O(n)
let enriched = orders.map(order => ({
  ...order,
  customer_name: customerMap[order.customer_id]?.name
}));

3. Avoid Unnecessary Transformations

Only transform data that actually needs transformation:

let products = job.data.data;

let processed = products.map(product => {
  let result = { ...product };
  
  // Only calculate if price changed
  if (product.price !== product.previous_price) {
    result.discount_percentage = calculateDiscount(product);
  }
  
  // Only enrich if category is missing
  if (!product.category_name) {
    result.category_name = lookupCategory(product.category_id);
  }
  
  return result;
});

4. Batch API Requests

If the destination API supports bulk operations, use them instead of individual requests:

let orders = job.data.data;

// Bad: 1 API call per order
for (let order of orders) {
  await api.post('/destination/orders', order);
}
// 1000 orders = 1000 API calls

let orders = job.data.data;

// Good: Batch 100 orders per call
for (let i = 0; i < orders.length; i += 100) {
  let batch = orders.slice(i, i + 100);
  await api.post('/destination/orders/batch', { orders: batch });
}
// 1000 orders = 10 API calls

Network & Connection Optimization

1. Use Private Networks for On-Premise Systems

Tailscale VPN connections are significantly faster than routing through public internet:

Learn more about Private Networks

2. Connection Pooling for Databases

Reuse database connections instead of creating new ones for each query:

// Connection pooling (handled by database client)
const mysql = require('mysql2/promise');

const pool = mysql.createPool({
  host: 'database.local',
  user: 'user',
  password: 'pass',
  database: 'mydb',
  waitForConnections: true,
  connectionLimit: 10, // Reuse up to 10 connections
  queueLimit: 0
});

const [rows] = await pool.query('SELECT * FROM orders');

3. Compression for Large Payloads

Enable compression when transferring large amounts of data:

// Enable compression for API requests
const response = await fetch('https://api.example.com/large-data', {
  headers: {
    'Accept-Encoding': 'gzip, deflate, br',
    'Content-Type': 'application/json'
  }
});

// Data is automatically decompressed

Caching Strategies

Cache Static or Rarely-Changed Data

Don't fetch data that doesn't change frequently:

// Cache static data that rarely changes
let cache = {
  categories: null,
  lastFetch: null
};

function getCategories() {
  // Cache for 24 hours
  const CACHE_DURATION = 24 * 60 * 60 * 1000;
  
  if (cache.categories && 
      cache.lastFetch && 
      (Date.now() - cache.lastFetch) < CACHE_DURATION) {
    console.log('Using cached categories');
    return cache.categories;
  }
  
  console.log('Fetching fresh categories');
  cache.categories = await api.get('/categories');
  cache.lastFetch = Date.now();
  
  return cache.categories;
}

// Use cached categories
let categories = getCategories();

Scheduling Optimization

1. Choose Appropriate Frequency

Not all integrations need to run every 5 minutes:

2. Avoid Peak Hours

Schedule resource-intensive integrations during off-peak hours:

• ✅ Good: Run full product catalog sync at 2 AM when traffic is low
• ✅ Good: Schedule bulk customer imports on weekends
• ❌ Bad: Running heavy integrations during business hours

3. Stagger Multiple Integrations

If you have multiple integrations, spread them out instead of running all at once:

Monitoring & Identifying Bottlenecks

Key Performance Metrics

Using Console Logs for Debugging

Add timing logs to identify slow sections:

let startTime = Date.now();
console.log('Starting order fetch...');

let orders = await api.get('/orders');
console.log(`Fetched ${orders.length} orders in ${Date.now() - startTime}ms`);

let transformStart = Date.now();
let processed = orders.map(transformOrder);
console.log(`Transformed data in ${Date.now() - transformStart}ms`);

let pushStart = Date.now();
await api.post('/destination/orders', processed);
console.log(`Pushed to destination in ${Date.now() - pushStart}ms`);

console.log(`Total execution time: ${Date.now() - startTime}ms`);

Performance Checklist

Real-World Performance Improvements

When to Optimize

Focus optimization efforts where they'll have the most impact:

Next Steps

• Error Handling - Implement retry logic and notifications for failed jobs
• Field Mapping - Use efficient field mapping instead of code
• Rewrite Functions - Write performant transformation code
• Webhooks - Replace scheduled polls with real-time webhooks