Batch Operations

Batch operations allow you to read or write multiple items in a single request, significantly improving performance and reducing API calls compared to individual operations. This guide covers batch get and batch write operations with best practices for handling unprocessed items.

Overview

DynamoDB provides two batch operations:

BatchGetItem:

Retrieve up to 100 items or 16MB of data
Read from one or more tables
Items retrieved in parallel
Supports projection expressions and consistent reads

BatchWriteItem:

Put or delete up to 25 items
Write to one or more tables
Operations processed in parallel
No conditional expressions supported (use Transactions if you need conditions)

Quick Start

The batch API uses static entry points and reuses existing request builders:

using Amazon.DynamoDBv2;
using Oproto.FluentDynamoDb;

var client = new AmazonDynamoDBClient();
var userTable = new UsersTable(client, "users");
var orderTable = new OrdersTable(client, "orders");

// Batch write - put multiple items
await DynamoDbBatch.Write
    .Add(userTable.Put(user1))
    .Add(userTable.Put(user2))
    .Add(orderTable.Delete(orderId))
    .ExecuteAsync();

// Batch get with tuple destructuring
var (user, order) = await DynamoDbBatch.Get
    .Add(userTable.Get(userId))
    .Add(orderTable.Get(orderId))
    .ExecuteAndMapAsync<User, Order>();

Important: Always check for unprocessed items in the response. DynamoDB may not process all items due to capacity limits.

Batch Write Operations

Batch write operations put or delete multiple items in a single request using the DynamoDbBatch.Write entry point.

Basic Batch Put

var users = new List<User>
{
    new User { UserId = "user1", Name = "Alice", Email = "alice@example.com" },
    new User { UserId = "user2", Name = "Bob", Email = "bob@example.com" },
    new User { UserId = "user3", Name = "Charlie", Email = "charlie@example.com" }
};

await DynamoDbBatch.Write
    .Add(userTable.Put(users[0]))
    .Add(userTable.Put(users[1]))
    .Add(userTable.Put(users[2]))
    .ExecuteAsync();

Basic Batch Delete

await DynamoDbBatch.Write
    .Add(userTable.Delete("user1"))
    .Add(userTable.Delete("user2"))
    .Add(userTable.Delete("user3"))
    .ExecuteAsync();

Delete with Composite Keys

For tables with both partition key and sort key:

await DynamoDbBatch.Write
    .Add(orderTable.Delete("customer123", "order1"))
    .Add(orderTable.Delete("customer123", "order2"))
    .Add(orderTable.Delete("customer456", "order3"))
    .ExecuteAsync();

Mixed Put and Delete Operations

Combine put and delete operations in a single batch:

await DynamoDbBatch.Write
    // Add new users
    .Add(userTable.Put(newUser1))
    .Add(userTable.Put(newUser2))
    
    // Delete old users
    .Add(userTable.Delete("oldUser1"))
    .Add(userTable.Delete("oldUser2"))
    .ExecuteAsync();

Batch Write to Multiple Tables

Write to different tables in a single batch request:

await DynamoDbBatch.Write
    .Add(userTable.Put(user))
    .Add(orderTable.Put(order))
    .Add(auditTable.Put(auditEntry))
    .ExecuteAsync();

Note: Batch writes do not support conditional expressions. If you need conditions, use Transactions instead.

Batch Get Operations

Batch get operations retrieve multiple items efficiently in a single request using the DynamoDbBatch.Get entry point.

Basic Batch Get

var response = await DynamoDbBatch.Get
    .Add(userTable.Get("user1"))
    .Add(userTable.Get("user2"))
    .Add(userTable.Get("user3"))
    .ExecuteAsync();

// Deserialize items
var users = response.GetItemsRange<User>(0, 2);
foreach (var user in users)
{
    if (user != null)
    {
        Console.WriteLine($"User: {user.Name}");
    }
}

ExecuteAndMapAsync - Tuple Destructuring

For convenience with small numbers of items, use ExecuteAndMapAsync to get a tuple:

// 2 items
var (user, order) = await DynamoDbBatch.Get
    .Add(userTable.Get(userId))
    .Add(orderTable.Get(orderId))
    .ExecuteAndMapAsync<User, Order>();

// 3 items
var (user, account, order) = await DynamoDbBatch.Get
    .Add(userTable.Get(userId))
    .Add(accountTable.Get(accountId))
    .Add(orderTable.Get(orderId))
    .ExecuteAndMapAsync<User, Account, Order>();

// Up to 8 items supported
var (item1, item2, item3, item4, item5, item6, item7, item8) = 
    await DynamoDbBatch.Get
        .Add(table1.Get(key1))
        .Add(table2.Get(key2))
        // ... up to 8 items
        .ExecuteAndMapAsync<T1, T2, T3, T4, T5, T6, T7, T8>();

Response Deserialization Methods

The BatchGetResponse provides multiple ways to deserialize items:

var response = await DynamoDbBatch.Get
    .Add(userTable.Get("user1"))
    .Add(userTable.Get("user2"))
    .Add(userTable.Get("user3"))
    .Add(orderTable.Get("order1"))
    .ExecuteAsync();

// Get single item by index
var user1 = response.GetItem<User>(0);

// Get multiple items of same type by indices
var users = response.GetItems<User>(0, 1, 2);

// Get contiguous range of items
var allUsers = response.GetItemsRange<User>(0, 2); // indices 0, 1, 2

// Get item from different table
var order = response.GetItem<Order>(3);

// Check total count
Console.WriteLine($"Retrieved {response.Count} items");

Batch Get with Composite Keys

var response = await DynamoDbBatch.Get
    .Add(orderTable.Get("customer123", "order1"))
    .Add(orderTable.Get("customer123", "order2"))
    .Add(orderTable.Get("customer456", "order3"))
    .ExecuteAsync();

var orders = response.GetItemsRange<Order>(0, 2);

Batch Get with Projection

Retrieve only specific attributes to reduce data transfer:

var response = await DynamoDbBatch.Get
    .Add(userTable.Get("user1").WithProjection("name, email"))
    .Add(userTable.Get("user2").WithProjection("name, email"))
    .Add(userTable.Get("user3").WithProjection("name, email"))
    .ExecuteAsync();

var users = response.GetItemsRange<User>(0, 2);

Projection Benefits:

Reduces network bandwidth
Lowers read capacity consumption
Improves response time for large items

Batch Get with Consistent Reads

var response = await DynamoDbBatch.Get
    .Add(userTable.Get("user1").UsingConsistentRead())
    .Add(userTable.Get("user2").UsingConsistentRead())
    .ExecuteAsync();

var users = response.GetItemsRange<User>(0, 1);

Note: Consistent reads consume twice the read capacity. Use them only when you need the most up-to-date data.

Batch Get from Multiple Tables

var response = await DynamoDbBatch.Get
    .Add(userTable.Get("user123"))
    .Add(userTable.Get("user456"))
    .Add(orderTable.Get("customer123", "order1"))
    .Add(productTable.Get("prod789"))
    .ExecuteAsync();

// Items are returned in the order they were added
var user1 = response.GetItem<User>(0);
var user2 = response.GetItem<User>(1);
var order = response.GetItem<Order>(2);
var product = response.GetItem<Product>(3);

Handling Unprocessed Items

DynamoDB may not process all items in a batch request due to capacity limits or other constraints. Always check for and handle unprocessed items.

Checking for Unprocessed Items

// Batch get
var getResponse = await DynamoDbBatch.Get
    .Add(userTable.Get("user1"))
    .Add(userTable.Get("user2"))
    .ExecuteAsync();

if (getResponse.HasUnprocessedKeys)
{
    Console.WriteLine($"Unprocessed keys in {getResponse.UnprocessedKeys.Count} tables");
    // Implement retry logic
}

// Batch write
var writeResponse = await DynamoDbBatch.Write
    .Add(userTable.Put(user1))
    .Add(userTable.Put(user2))
    .ExecuteAsync();

if (writeResponse.UnprocessedItems.Count > 0)
{
    Console.WriteLine($"Unprocessed items in {writeResponse.UnprocessedItems.Count} tables");
    // Implement retry logic
}

Retry Logic with Exponential Backoff

public async Task<BatchGetResponse> BatchGetWithRetry(
    BatchGetBuilder builder,
    int maxRetries = 3)
{
    var response = await builder.ExecuteAsync();
    var retryCount = 0;
    
    while (response.HasUnprocessedKeys && retryCount < maxRetries)
    {
        // Exponential backoff: 100ms, 200ms, 400ms
        var delayMs = 100 * (int)Math.Pow(2, retryCount);
        await Task.Delay(delayMs);
        
        Console.WriteLine($"Retry {retryCount + 1}: unprocessed keys remaining");
        
        // Retry with unprocessed keys
        var retryRequest = new BatchGetItemRequest
        {
            RequestItems = response.UnprocessedKeys
        };
        
        var retryResponse = await client.BatchGetItemAsync(retryRequest);
        response = new BatchGetResponse(retryResponse, tableOrder);
        retryCount++;
    }
    
    if (response.HasUnprocessedKeys)
    {
        Console.WriteLine($"Failed to process all items after {maxRetries} retries");
    }
    
    return response;
}

Chunking Large Batches

When you have more items than the batch limit allows, split them into chunks.

Chunking Batch Writes (25 items per batch)

public async Task BatchWriteInChunks<T>(
    List<T> items,
    Func<T, PutItemRequestBuilder<T>> putBuilder,
    int chunkSize = 25)
{
    // Split into chunks of 25 (BatchWriteItem limit)
    for (int i = 0; i < items.Count; i += chunkSize)
    {
        var chunk = items.Skip(i).Take(chunkSize).ToList();
        
        var batch = DynamoDbBatch.Write;
        foreach (var item in chunk)
        {
            batch.Add(putBuilder(item));
        }
        
        var response = await batch.ExecuteAsync();
        
        // Handle unprocessed items
        if (response.UnprocessedItems.Count > 0)
        {
            Console.WriteLine($"Chunk {i / chunkSize + 1}: unprocessed items");
            // Implement retry logic
        }
    }
}

// Usage
await BatchWriteInChunks(
    allUsers,
    user => userTable.Put(user)
);

Parallel Batch Operations

For better throughput with large datasets:

public async Task ParallelBatchWrite<T>(
    List<T> items,
    Func<T, PutItemRequestBuilder<T>> putBuilder,
    int maxParallel = 4)
{
    // Split into chunks
    var chunks = items
        .Select((item, index) => new { item, index })
        .GroupBy(x => x.index / 25)
        .Select(g => g.Select(x => x.item).ToList())
        .ToList();
    
    // Process chunks in parallel (with limit)
    var semaphore = new SemaphoreSlim(maxParallel);
    var tasks = chunks.Select(async chunk =>
    {
        await semaphore.WaitAsync();
        try
        {
            var batch = DynamoDbBatch.Write;
            foreach (var item in chunk)
            {
                batch.Add(putBuilder(item));
            }
            await batch.ExecuteAsync();
        }
        finally
        {
            semaphore.Release();
        }
    });
    
    await Task.WhenAll(tasks);
}

Best Practices

Always Handle Unprocessed Items

// ✅ Good - handles unprocessed items
var response = await DynamoDbBatch.Write
    .Add(userTable.Put(user))
    .ExecuteAsync();
    
if (response.UnprocessedItems.Count > 0)
{
    // Retry with exponential backoff
}

// ❌ Avoid - ignores unprocessed items
await DynamoDbBatch.Write.Add(userTable.Put(user)).ExecuteAsync();

Use Projection Expressions

// ✅ Good - only retrieve needed attributes
await DynamoDbBatch.Get
    .Add(userTable.Get("user123").WithProjection("name, email"))
    .ExecuteAsync();

// ❌ Avoid - retrieves all attributes
await DynamoDbBatch.Get
    .Add(userTable.Get("user123"))
    .ExecuteAsync();

Chunk Large Batches

// ✅ Good - chunks into batches of 25
await BatchWriteInChunks(allUsers, user => userTable.Put(user), 25);

// ❌ Avoid - trying to write more than 25 items
var batch = DynamoDbBatch.Write;
foreach (var user in allUsers) // Could be > 25 items
{
    batch.Add(userTable.Put(user));
}
await batch.ExecuteAsync(); // Will throw ValidationException

Use Batch Operations for Bulk Reads/Writes

// ✅ Good - single batch request
await DynamoDbBatch.Get
    .Add(userTable.Get("user1"))
    .Add(userTable.Get("user2"))
    .Add(userTable.Get("user3"))
    .ExecuteAsync();

// ❌ Avoid - multiple individual requests
foreach (var userId in userIds)
{
    await userTable.Get(userId).GetItemAsync();
}

Implement Exponential Backoff

// ✅ Good - exponential backoff for retries
var delayMs = 100 * (int)Math.Pow(2, retryCount);
await Task.Delay(delayMs);

// ❌ Avoid - fixed delay or immediate retry
await Task.Delay(100); // Same delay every time