Automatic Storage

Store large data externally (e.g., S3) with only a reference in DynamoDB using the [BlobStorage] attribute and BlobData<T> wrapper type.

note

The [BlobReference] attribute is deprecated. Use [BlobStorage] with BlobData<T> wrapper type instead.

Overview

The [BlobStorage] attribute enables automatic external storage for large data that exceeds DynamoDB's 400KB item limit. When you save an entity, the blob data is automatically uploaded to your configured storage provider (e.g., S3), and only a reference key is stored in DynamoDB.

BlobData<T> Wrapper Type

The BlobData<T> wrapper encapsulates blob storage behavior:

Property/Method	Description
Value	Gets the loaded data. Throws InvalidOperationException if not loaded.
ReferenceKey	Gets the storage key, or null if not yet stored.
IsLoaded	Returns true when data has been loaded from storage.
HasPendingData	Returns true when instance has new data to be stored.
Create(T value)	Static factory to create instance with data to be stored.
LoadAsync()	Loads data from storage. Idempotent - returns immediately if already loaded.

Basic Usage

1. Define Entity with BlobStorage

using Oproto.FluentDynamoDb.Attributes;
using Oproto.FluentDynamoDb.Providers.BlobStorage;

[DynamoDbTable("files")]
public partial class FileMetadata
{
    [PartitionKey]
    [DynamoDbAttribute("file_id")]
    public string FileId { get; set; } = string.Empty;
    
    [BlobStorage]
    [DynamoDbAttribute("data")]
    public BlobData<byte[]> Data { get; set; } = default!;
}

2. Configure Blob Storage Provider

using Oproto.FluentDynamoDb;
using Oproto.FluentDynamoDb.BlobStorage.S3;

var s3Client = new AmazonS3Client();
var blobProvider = new S3BlobProvider(s3Client, "my-files-bucket", "uploads");

var options = new FluentDynamoDbOptions()
    .WithBlobStorage(blobProvider);

var table = new FileTable(dynamoDbClient, "files", options);

3. Save Entity with Blob Data

Use BlobData<T>.Create() to create instances with data to be stored:

var file = new FileMetadata
{
    FileId = "file-123",
    Data = BlobData<byte[]>.Create(File.ReadAllBytes("large-file.pdf"))
};

await table.Files.PutAsync(file);

4. Load Entity and Access Data

var loaded = await table.Files.GetAsync("file-123");

// Data is already loaded (eager loading is the default)
var data = loaded.Data.Value;

Lazy vs Eager Loading

By default, blob data is loaded eagerly during entity deserialization. For large blobs that aren't always needed, use lazy loading.

Eager Loading (Default)

Data is automatically loaded when the entity is retrieved:

[BlobStorage]
[DynamoDbAttribute("data")]
public BlobData<byte[]> Data { get; set; } = default!;

// Usage
var entity = await table.Files.GetAsync("file-123");
var data = entity.Data.Value;  // Already loaded

Lazy Loading

Data is loaded only when LoadAsync() is called:

[BlobStorage(LazyLoad = true)]
[DynamoDbAttribute("thumbnail")]
public BlobData<byte[]>? Thumbnail { get; set; }

// Usage
var entity = await table.Files.GetAsync("file-123");

// Check if loaded before accessing
if (entity.Thumbnail != null && !entity.Thumbnail.IsLoaded)
{
    await entity.Thumbnail.LoadAsync();
}

var thumbnail = entity.Thumbnail?.Value;

When to Use Lazy Loading

• Large blobs that aren't always needed
• Multiple blob properties where you only need some
• Performance-critical paths where you want to defer loading
• Reducing initial load time for entities with many blobs

Blob Storage Strategies

Configure how failures between blob storage and DynamoDB operations are handled.

BestEffortCleanupStrategy (Default)

Attempts to clean up orphaned blobs when DynamoDB operations fail:

var options = new FluentDynamoDbOptions()
    .WithBlobStorage(blobProvider);  // Uses BestEffortCleanupStrategy by default

// Or explicitly:
var options = new FluentDynamoDbOptions()
    .WithBlobStorage(blobProvider)
    .WithBlobStorageStrategy(new BestEffortCleanupStrategy(blobProvider, logger));

Behavior:

• Uploads blobs to S3 before DynamoDB write
• If DynamoDB write fails, attempts to delete uploaded blobs (best effort)
• Cleanup failures are logged but don't throw exceptions
• On successful DynamoDB delete, deletes associated blobs from S3

NoCleanupStrategy

Simple strategy for non-critical data where orphaned blobs are acceptable:

var options = new FluentDynamoDbOptions()
    .WithBlobStorage(blobProvider)
    .WithBlobStorageStrategy(new NoCleanupStrategy(blobProvider));

Behavior:

• Uploads blobs to S3 before DynamoDB write
• No cleanup on DynamoDB write failure (orphaned blobs may remain)
• No blob deletion on DynamoDB delete

Choosing a Strategy

Strategy	Use Case
BestEffortCleanupStrategy	Production workloads where you want to minimize orphaned blobs
NoCleanupStrategy	Development, testing, or non-critical data where simplicity is preferred

Attribute Combinations

The [BlobStorage] attribute can be combined with other attributes for additional functionality.

BlobStorage + JsonBlob

Serialize complex objects to JSON before uploading to S3:

[DynamoDbTable("documents")]
public partial class LargeDocument
{
    [PartitionKey]
    [DynamoDbAttribute("doc_id")]
    public string DocumentId { get; set; } = string.Empty;
    
    [BlobStorage]
    [JsonBlob]
    [DynamoDbAttribute("content")]
    public BlobData<ComplexContent> Content { get; set; } = default!;
}

public class ComplexContent
{
    public string Title { get; set; } = string.Empty;
    public List<Section> Sections { get; set; } = new();
    public Dictionary<string, string> Metadata { get; set; } = new();
}

Processing order:

• Save: Object → JSON serialize → Upload to S3 → Store reference in DynamoDB
• Load: Read reference from DynamoDB → Download from S3 → JSON deserialize → Object

tip

Remember to configure a JSON serializer when using [JsonBlob]:

var options = new FluentDynamoDbOptions()
    .WithBlobStorage(blobProvider)
    .WithSystemTextJson();  // or .WithNewtonsoftJson()

BlobStorage + Encrypted

Encrypt data before uploading to S3:

[DynamoDbTable("secrets")]
public partial class SecretDocument
{
    [PartitionKey]
    [DynamoDbAttribute("id")]
    public string Id { get; set; } = string.Empty;
    
    [BlobStorage]
    [Encrypted]
    [DynamoDbAttribute("secret")]
    public BlobData<byte[]> SecretData { get; set; } = default!;
}

Requires encryption configuration:

var options = new FluentDynamoDbOptions()
    .WithBlobStorage(blobProvider)
    .WithEncryption(fieldEncryptor);

BlobStorage + Sensitive

Redact blob reference keys and values in logs:

[BlobStorage]
[Sensitive]
[DynamoDbAttribute("pii")]
public BlobData<byte[]> PersonalData { get; set; } = default!;

All Three Combined

// JSON serialize → Encrypt → Upload to S3
[BlobStorage]
[JsonBlob]
[Encrypted]
[DynamoDbAttribute("encrypted_content")]
public BlobData<SensitiveContent> EncryptedContent { get; set; } = default!;

Error Handling

Scenario	Exception
Access Value before LoadAsync() on lazy-loaded blob	InvalidOperationException
LoadAsync() without provider configured	InvalidOperationException
S3 upload/download failure	BlobStorageException
[Encrypted] without encryptor configured	EncryptionRequiredException
[BlobStorage] without provider configured	InvalidOperationException

Example Error Handling

try
{
    var entity = await table.Files.GetAsync("file-123");
    
    if (entity.Thumbnail != null)
    {
        await entity.Thumbnail.LoadAsync();
        var data = entity.Thumbnail.Value;
    }
}
catch (BlobStorageException ex)
{
    // Handle S3 errors (network, permissions, etc.)
    logger.LogError(ex, "Failed to load blob: {Key}", ex.BlobKey);
}
catch (InvalidOperationException ex)
{
    // Handle configuration errors
    logger.LogError(ex, "Blob storage not configured correctly");
}

Migration from BlobReference

If you're migrating from the deprecated [BlobReference] attribute:

// Before (deprecated):
[BlobReference(BlobProvider.S3)]
[DynamoDbAttribute("data")]
public byte[] Data { get; set; }

// After:
[BlobStorage]
[DynamoDbAttribute("data")]
public BlobData<byte[]> Data { get; set; } = default!;

Code Changes

// Creating data:
// Before: entity.Data = bytes;
// After:  entity.Data = BlobData<byte[]>.Create(bytes);

// Accessing data:
// Before: var bytes = entity.Data;
// After:  var bytes = entity.Data.Value;

See Also

• S3 Integration - Configure S3 as your blob storage provider
• Reference Management - Manage blob references in DynamoDB
• Encryption Overview - Encrypt blob data