Skip to main content

S3 Integration

Store large binary data in S3 while keeping references in DynamoDB.

Overview​

FluentDynamoDB integrates with Amazon S3 to store large data that exceeds DynamoDB's 400KB item limit. The S3BlobProvider handles all upload and download operations automatically.

Installation​

dotnet add package Oproto.FluentDynamoDb.BlobStorage.S3

Setup​

1. Create S3 Client and Provider​

using Amazon.S3;
using Oproto.FluentDynamoDb.BlobStorage.S3;

var s3Client = new AmazonS3Client();
var blobProvider = new S3BlobProvider(
    s3Client, 
    bucketName: "my-files-bucket", 
    keyPrefix: "uploads"  // Optional prefix for all keys
);

2. Configure FluentDynamoDbOptions​

using Oproto.FluentDynamoDb;

var options = new FluentDynamoDbOptions()
    .WithBlobStorage(blobProvider);

var table = new FileTable(dynamoDbClient, "files", options);

Configuration Options​

Key Prefix​

Organize blobs under a common prefix:

// All blobs stored under "uploads/" prefix
var blobProvider = new S3BlobProvider(s3Client, "my-bucket", "uploads");

// Blobs stored under "tenant-123/files/" prefix
var blobProvider = new S3BlobProvider(s3Client, "my-bucket", "tenant-123/files");

Custom Key Generation​

By default, keys are generated using GUIDs. You can customize this:

var blobProvider = new S3BlobProvider(s3Client, "my-bucket", "uploads")
    .WithKeyGenerator((entityType, propertyName) => 
        $"{entityType}/{propertyName}/{Guid.NewGuid()}");

Entity Definition​

Use [BlobStorage] attribute with BlobData<T> wrapper:

using Oproto.FluentDynamoDb.Attributes;
using Oproto.FluentDynamoDb.Providers.BlobStorage;

[DynamoDbTable("files")]
public partial class FileMetadata
{
    [PartitionKey]
    [DynamoDbAttribute("file_id")]
    public string FileId { get; set; } = string.Empty;
    
    [DynamoDbAttribute("filename")]
    public string Filename { get; set; } = string.Empty;
    
    [BlobStorage]
    [DynamoDbAttribute("data")]
    public BlobData<byte[]> Data { get; set; } = default!;
    
    [BlobStorage(LazyLoad = true)]
    [DynamoDbAttribute("thumbnail")]
    public BlobData<byte[]>? Thumbnail { get; set; }
}

Usage Examples​

Saving Files​

var file = new FileMetadata
{
    FileId = "file-123",
    Filename = "document.pdf",
    Data = BlobData<byte[]>.Create(File.ReadAllBytes("document.pdf")),
    Thumbnail = BlobData<byte[]>.Create(thumbnailBytes)
};

await table.Files.PutAsync(file);

Loading Files​

var file = await table.Files.GetAsync("file-123");

// Eager-loaded data is immediately available
var pdfData = file.Data.Value;

// Lazy-loaded data requires explicit loading
if (file.Thumbnail != null && !file.Thumbnail.IsLoaded)
{
    await file.Thumbnail.LoadAsync();
    var thumbnailData = file.Thumbnail.Value;
}

Updating Files​

var file = await table.Files.GetAsync("file-123");

// Replace blob data
file.Data = BlobData<byte[]>.Create(newFileBytes);

await table.Files.PutAsync(file);
// Old blob is cleaned up based on configured strategy

IAM Permissions​

Ensure your application has the necessary S3 permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::my-files-bucket/*"
        }
    ]
}

Error Handling​

try
{
    await table.Files.PutAsync(file);
}
catch (BlobStorageException ex)
{
    // S3 operation failed
    logger.LogError(ex, "Failed to store blob: {Key}", ex.BlobKey);
}

See Also​