Reliability

Design your webhook integration to handle the realities of distributed systems: retries, out-of-order delivery, and duplicate events.

Implement Idempotency

Use the eventId field to ensure you process each event exactly once, even if Smartcar retries delivery or you reprocess from your queue.

Why Idempotency Matters

Smartcar retries failed deliveries with the same eventId
Your queue worker might process the same message multiple times
Prevents duplicate database updates or notifications
Enables safe reprocessing of historical events

Implementation Strategies

Redis
DynamoDB
PostgreSQL

import redis

redis_client = redis.Redis()

def process_webhook(payload):
    event_id = payload.get("eventId")
    
    # Check if already processed
    if redis_client.exists(f"processed:{event_id}"):
        print(f"Already processed {event_id}, skipping")
        return
    
    # Process the event
    process_vehicle_data(payload)
    
    # Mark as processed (expires after 7 days)
    redis_client.setex(
        f"processed:{event_id}",
        time=604800,  # 7 days
        value="1"
    )

const processWebhook = async (payload) => {
  const { eventId } = payload;
  
  // Check if already processed
  const existing = await dynamodb.get({
    TableName: 'ProcessedEvents',
    Key: { eventId }
  }).promise();
  
  if (existing.Item) {
    console.log(`Already processed ${eventId}, skipping`);
    return;
  }
  
  // Process the event
  await processVehicleData(payload);
  
  // Mark as processed with TTL
  await dynamodb.put({
    TableName: 'ProcessedEvents',
    Item: {
      eventId,
      processedAt: new Date().toISOString(),
      ttl: Math.floor(Date.now() / 1000) + 604800 // 7 days
    }
  }).promise();
};

def process_webhook(payload):
    event_id = payload.get("eventId")
    
    # Use unique constraint to prevent duplicates
    try:
        db.execute(
            "INSERT INTO processed_events (event_id, processed_at) VALUES (%s, NOW())",
            (event_id,)
        )
    except IntegrityError:
        # Already processed
        print(f"Already processed {event_id}, skipping")
        return
    
    # Process the event
    process_vehicle_data(payload)

-- Table schema
CREATE TABLE processed_events (
    event_id VARCHAR(255) PRIMARY KEY,
    processed_at TIMESTAMP NOT NULL
);

-- Add TTL-style cleanup (run periodically)
DELETE FROM processed_events 
WHERE processed_at < NOW() - INTERVAL '7 days';

Retention period: Store processed eventId values for at least 7 days to handle all retries and late reprocessing scenarios.

Handle Out-of-Order Delivery

Webhook events are delivered concurrently and may arrive out of order. Never assume events arrive in chronological sequence.

Use Timestamps for Freshness

Always check if incoming data is newer than your current stored state:

def update_vehicle_state(payload):
    vehicle_id = payload.get("vehicleId")
    delivered_at = payload["meta"]["deliveredAt"]
    
    # Get current stored state
    current = db.get_vehicle_state(vehicle_id)
    
    # Only update if this event is newer
    if current and current.updated_at > delivered_at:
        print(f"Ignoring older event for {vehicle_id}")
        return
    
    # Safe to update
    db.update_vehicle_state(vehicle_id, payload["data"], delivered_at)

Why Order Matters

Network Delays

Events sent at different times may experience different network latencies, causing them to arrive out of sequence.

Retry Timing

If an older event fails initially and is retried later, it might arrive after newer events that succeeded on first attempt.

Concurrent Delivery

Smartcar delivers events concurrently for performance. Events sent milliseconds apart might arrive in reverse order.

Timestamp-Based Updates

def update_signal_value(vehicle_id, signal_path, value, timestamp):
    """Update signal only if timestamp is newer"""
    with db.transaction():
        current = db.query(
            "SELECT value, updated_at FROM signals WHERE vehicle_id = %s AND path = %s",
            (vehicle_id, signal_path)
        )
        
        if current and current['updated_at'] >= timestamp:
            # Existing value is newer or same age
            return False
        
        # Update with newer value
        db.execute(
            "INSERT INTO signals (vehicle_id, path, value, updated_at) VALUES (%s, %s, %s, %s) ON CONFLICT (vehicle_id, path) DO UPDATE SET value = EXCLUDED.value, updated_at = EXCLUDED.updated_at WHERE signals.updated_at < EXCLUDED.updated_at",
            (vehicle_id, signal_path, value, timestamp)
        )
        return True

Handle Retries Gracefully

Smartcar automatically retries failed deliveries up to 3 times with exponential backoff.

Retry Identification

Each delivery attempt receives a unique deliveryId, but the eventId remains constant:

{
  "eventId": "abc-123",         // Same across all retries
  "meta": {
    "deliveryId": "xyz-789",    // Unique per attempt
    "deliveredAt": "2025-01-15T10:30:45Z"
  }
}

Processing Strategy

Check for duplicate eventId

Use idempotency check to skip already-processed events

Process the payload

Perform your business logic

Mark as processed

Store the eventId to prevent reprocessing

Return 200

Acknowledge successful processing

Don’t trigger retries manually. If you return a 2xx status code and then discover an issue, you cannot ask Smartcar to retry. The delivery is considered successful.

Transactional Processing

Ensure database updates and idempotency tracking happen atomically:

async function processWebhook(payload) {
  const { eventId } = payload;
  
  const client = await pool.connect();
  try {
    await client.query('BEGIN');
    
    // Try to insert processed event
    const result = await client.query(
      `INSERT INTO processed_events (event_id, processed_at) 
       VALUES ($1, NOW()) 
       ON CONFLICT (event_id) DO NOTHING 
       RETURNING event_id`,
      [eventId]
    );
    
    if (result.rowCount === 0) {
      // Already processed
      await client.query('ROLLBACK');
      return;
    }
    
    // Process within same transaction
    await updateVehicleData(client, payload);
    await sendNotifications(client, payload);
    
    await client.query('COMMIT');
  } catch (error) {
    await client.query('ROLLBACK');
    throw error;
  } finally {
    client.release();
  }
}

Recovery Strategies

Dead Letter Queue

Route persistently failing events to a dead letter queue for manual investigation:

async function processWebhook(payload) {
  const maxRetries = 3;
  let retryCount = 0;
  
  while (retryCount < maxRetries) {
    try {
      await doProcessing(payload);
      return; // Success
    } catch (error) {
      retryCount++;
      if (retryCount >= maxRetries) {
        // Move to DLQ
        await dlq.send({
          payload,
          error: error.message,
          attempts: retryCount
        });
      } else {
        // Wait before retry
        await sleep(Math.pow(2, retryCount) * 1000);
      }
    }
  }
}

Circuit Breaker

Stop processing if downstream dependencies are failing:

class CircuitBreaker:
    def __init__(self, failure_threshold=5):
        self.failure_count = 0
        self.failure_threshold = failure_threshold
        self.is_open = False
        self.last_failure_time = None
    
    def call(self, func, *args):
        if self.is_open:
            # Check if we should try again
            if time.time() - self.last_failure_time > 60:
                self.is_open = False
                self.failure_count = 0
            else:
                raise Exception("Circuit breaker is open")
        
        try:
            result = func(*args)
            self.failure_count = 0
            return result
        except Exception as e:
            self.failure_count += 1
            self.last_failure_time = time.time()
            
            if self.failure_count >= self.failure_threshold:
                self.is_open = True
            
            raise e

# Usage
breaker = CircuitBreaker()

def process_webhook(payload):
    try:
        breaker.call(update_database, payload)
    except Exception:
        # Return 503 to trigger Smartcar retry later
        return {"error": "Service unavailable"}, 503

Next Steps

Monitoring

Track idempotency hits and processing failures

VEHICLE_ERROR Events

Handle error notifications and resolution tracking

Delivery Behavior

Understanding retry policies

Testing

Test retry scenarios

Introduction

Webhooks

Vehicle Authorization & Permissions

Vehicle Data

Vehicle Commands

Management API

Compatibility API

Implement Idempotency

Why Idempotency Matters

Implementation Strategies

Handle Out-of-Order Delivery

Use Timestamps for Freshness

Why Order Matters

Timestamp-Based Updates

Handle Retries Gracefully

Retry Identification

Processing Strategy

Transactional Processing

Recovery Strategies

Dead Letter Queue

Circuit Breaker

Next Steps

Monitoring

VEHICLE_ERROR Events

Delivery Behavior

Testing

Introduction

Webhooks

Vehicle Authorization & Permissions

Vehicle Data

Vehicle Commands

Management API

Compatibility API

​Implement Idempotency

​Why Idempotency Matters

​Implementation Strategies

​Handle Out-of-Order Delivery

​Use Timestamps for Freshness

​Why Order Matters

​Timestamp-Based Updates

​Handle Retries Gracefully

​Retry Identification

​Processing Strategy

​Transactional Processing

​Recovery Strategies

​Dead Letter Queue

​Circuit Breaker

​Next Steps

Monitoring

VEHICLE_ERROR Events

Delivery Behavior

Testing

Implement Idempotency

Why Idempotency Matters

Implementation Strategies

Handle Out-of-Order Delivery

Use Timestamps for Freshness

Why Order Matters

Timestamp-Based Updates

Handle Retries Gracefully

Retry Identification

Processing Strategy

Transactional Processing

Recovery Strategies

Dead Letter Queue

Circuit Breaker

Next Steps