Skip to main content

S3 Integration

Store large binary data in S3 while keeping references in DynamoDB.

Overview​

FluentDynamoDB integrates with Amazon S3 to store large data that exceeds DynamoDB's 400KB item limit. The S3BlobProvider handles all upload and download operations automatically.

Installation​

dotnet add package Oproto.FluentDynamoDb.BlobStorage.S3

Setup​

1. Create S3 Client and Provider​

using Amazon.S3;
using Oproto.FluentDynamoDb.BlobStorage.S3;

var s3Client = new AmazonS3Client();
var blobProvider = new S3BlobProvider(
    s3Client, 
    bucketName: "my-files-bucket", 
    keyPrefix: "uploads"  // Optional prefix for all keys
);

2. Configure FluentDynamoDbOptions​

using Oproto.FluentDynamoDb;

var options = new FluentDynamoDbOptions()
    .WithBlobStorage(blobProvider);

var table = new FileTable(dynamoDbClient, "files", options);

Configuration Options​

Key Prefix​

Organize blobs under a common prefix:

// All blobs stored under "uploads/" prefix
var blobProvider = new S3BlobProvider(s3Client, "my-bucket", "uploads");

// Blobs stored under "tenant-123/files/" prefix
var blobProvider = new S3BlobProvider(s3Client, "my-bucket", "tenant-123/files");

Custom Key Generation​

By default, keys are generated using GUIDs. You can customize this:

var blobProvider = new S3BlobProvider(s3Client, "my-bucket", "uploads")
    .WithKeyGenerator((entityType, propertyName) => 
        $"{entityType}/{propertyName}/{Guid.NewGuid()}");

Named Providers​

When your application stores blobs across multiple S3 buckets, named providers let you route individual properties to specific buckets. Properties without an explicit provider name continue to use the default provider.

Registering Named Providers​

Register a default provider and one or more named providers using WithBlobStorage:

var options = new FluentDynamoDbOptions()
    // Default provider — used when no Provider is specified on the attribute
    .WithBlobStorage(new S3BlobProvider(s3Client, "my-default-bucket", "data"))
    // Named provider for image storage
    .WithBlobStorage("images", new S3BlobProvider(s3Client, "my-images-bucket", "img"))
    // Named provider for document storage
    .WithBlobStorage("documents", new S3BlobProvider(s3Client, "my-docs-bucket", "docs"));

var table = new MediaTable(dynamoDbClient, "media", options);

Routing Properties to Named Providers​

Use the Provider property on the [BlobStorage] attribute to route a property to a specific named provider:

[DynamoDbTable("media")]
public partial class MediaItem
{
    [PartitionKey]
    [DynamoDbAttribute("pk")]
    public string Pk { get; set; } = string.Empty;

    [DynamoDbAttribute("title")]
    public string Title { get; set; } = string.Empty;

    // Routes to the "images" named provider → my-images-bucket
    [BlobStorage(Provider = "images")]
    [DynamoDbAttribute("photo")]
    public BlobData<byte[]> Photo { get; set; } = default!;

    // Routes to the "documents" named provider → my-docs-bucket
    [BlobStorage(Provider = "documents")]
    [DynamoDbAttribute("contract")]
    public BlobData<byte[]> Contract { get; set; } = default!;

    // No Provider specified → uses the default provider → my-default-bucket
    [BlobStorage]
    [DynamoDbAttribute("metadata_blob")]
    public BlobData<byte[]>? MetadataBlob { get; set; }
}

Complete Example​

A complete setup with multiple named providers routing different properties to different S3 buckets:

using Amazon.S3;
using Oproto.FluentDynamoDb;
using Oproto.FluentDynamoDb.BlobStorage.S3;

// Create provider instances for each bucket
var s3Client = new AmazonS3Client();
var defaultProvider = new S3BlobProvider(s3Client, "app-general-storage", "blobs");
var imageProvider = new S3BlobProvider(s3Client, "app-image-cdn", "originals");
var documentProvider = new S3BlobProvider(s3Client, "app-secure-docs", "contracts");

// Register all providers
var options = new FluentDynamoDbOptions()
    .WithBlobStorage(defaultProvider)
    .WithBlobStorage("images", imageProvider)
    .WithBlobStorage("documents", documentProvider);

var table = new MediaTable(new AmazonDynamoDBClient(), "media", options);

// Create and save an entity
var item = new MediaItem
{
    Pk = "media-001",
    Title = "Project Assets",
    Photo = BlobData<byte[]>.Create(photoBytes),         // → app-image-cdn/originals/...
    Contract = BlobData<byte[]>.Create(contractBytes),   // → app-secure-docs/contracts/...
    MetadataBlob = BlobData<byte[]>.Create(metaBytes)    // → app-general-storage/blobs/...
};

await table.Media.PutAsync(item);
note

Properties without a Provider value on the [BlobStorage] attribute always use the default provider registered via .WithBlobStorage(provider) (without a name). If no default provider is registered, those properties will throw at runtime.

Entity Definition​

Use [BlobStorage] attribute with BlobData<T> wrapper:

using Oproto.FluentDynamoDb.Attributes;
using Oproto.FluentDynamoDb.Providers.BlobStorage;

[DynamoDbTable("files")]
public partial class FileMetadata
{
    [PartitionKey]
    [DynamoDbAttribute("file_id")]
    public string FileId { get; set; } = string.Empty;
    
    [DynamoDbAttribute("filename")]
    public string Filename { get; set; } = string.Empty;
    
    [BlobStorage]
    [DynamoDbAttribute("data")]
    public BlobData<byte[]> Data { get; set; } = default!;
    
    [BlobStorage(LazyLoad = true)]
    [DynamoDbAttribute("thumbnail")]
    public BlobData<byte[]>? Thumbnail { get; set; }
}

Usage Examples​

Saving Files​

var file = new FileMetadata
{
    FileId = "file-123",
    Filename = "document.pdf",
    Data = BlobData<byte[]>.Create(File.ReadAllBytes("document.pdf")),
    Thumbnail = BlobData<byte[]>.Create(thumbnailBytes)
};

await table.Files.PutAsync(file);

Loading Files​

var file = await table.Files.GetAsync("file-123");

// Eager-loaded data is immediately available
var pdfData = file.Data.Value;

// Lazy-loaded data requires explicit loading
if (file.Thumbnail != null && !file.Thumbnail.IsLoaded)
{
    await file.Thumbnail.LoadAsync();
    var thumbnailData = file.Thumbnail.Value;
}

Updating Files​

var file = await table.Files.GetAsync("file-123");

// Replace blob data
file.Data = BlobData<byte[]>.Create(newFileBytes);

await table.Files.PutAsync(file);
// Old blob is cleaned up based on configured strategy

IAM Permissions​

Ensure your application has the necessary S3 permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::my-files-bucket/*"
        }
    ]
}

Error Handling​

try
{
    await table.Files.PutAsync(file);
}
catch (BlobStorageException ex)
{
    // S3 operation failed
    logger.LogError(ex, "Failed to store blob: {Key}", ex.BlobKey);
}

See Also​