Skip to main content

Reference Management

Manage S3 references stored in DynamoDB and understand how blob lifecycle is handled.

How References Work​

When you save an entity with [BlobStorage] properties:

  1. Upload Phase: Blob data is uploaded to S3
  2. Reference Storage: Only the S3 key is stored in DynamoDB
  3. Load Phase: When reading, the key is used to fetch data from S3
┌─────────────────┐     ┌─────────────────┐
│   DynamoDB      │     │      S3         │
├─────────────────┤     ├─────────────────┤
│ file_id: "123"  │     │ uploads/abc-def │
│ data: "abc-def" │────▶│ [binary data]   │
└─────────────────┘     └─────────────────┘

Reference Format​

The BlobData<T> wrapper provides access to reference information:

var file = await table.Files.GetAsync("file-123");

// Get the S3 key (reference)
string? key = file.Data.ReferenceKey;
// Example: "uploads/a1b2c3d4-e5f6-7890-abcd-ef1234567890"

// Check if data is loaded
bool isLoaded = file.Data.IsLoaded;

// Check if there's new data to store
bool hasPending = file.Data.HasPendingData;

Lifecycle Management​

Creating New Blobs​

// Create with pending data
var file = new FileMetadata
{
    FileId = "file-123",
    Data = BlobData<byte[]>.Create(bytes)  // HasPendingData = true
};

// After save, reference is assigned
await table.Files.PutAsync(file);
// file.Data.ReferenceKey is now set
// file.Data.HasPendingData = false

Updating Blobs​

When you replace blob data, the old blob may be cleaned up based on your configured strategy:

var file = await table.Files.GetAsync("file-123");
var oldKey = file.Data.ReferenceKey;  // "uploads/old-key"

// Replace with new data
file.Data = BlobData<byte[]>.Create(newBytes);

await table.Files.PutAsync(file);
// New blob uploaded, new reference stored
// Old blob cleanup depends on strategy

Deleting Entities​

When you delete an entity, associated blobs are handled by your cleanup strategy:

await table.Files.DeleteAsync("file-123");
// With BestEffortCleanupStrategy: S3 blob is deleted
// With NoCleanupStrategy: S3 blob remains (orphaned)

Cleanup Strategies​

BestEffortCleanupStrategy​

Automatically cleans up blobs when:

  • Entity is deleted
  • Blob property is replaced with new data
  • DynamoDB write fails after blob upload (rollback)
var options = new FluentDynamoDbOptions()
    .WithBlobStorage(blobProvider)
    .WithBlobStorageStrategy(new BestEffortCleanupStrategy(blobProvider, logger));

NoCleanupStrategy​

No automatic cleanup - blobs remain in S3:

var options = new FluentDynamoDbOptions()
    .WithBlobStorage(blobProvider)
    .WithBlobStorageStrategy(new NoCleanupStrategy(blobProvider));

Use this when:

  • You have separate S3 lifecycle policies
  • You need to retain blobs for audit purposes
  • Simplicity is more important than storage costs

Manual Reference Access​

For advanced scenarios, you can access the underlying reference:

var file = await table.Files.GetAsync("file-123");

if (file.Data.ReferenceKey != null)
{
    // Generate a presigned URL for direct S3 access
    var presignedUrl = s3Client.GetPreSignedURL(new GetPreSignedUrlRequest
    {
        BucketName = "my-bucket",
        Key = file.Data.ReferenceKey,
        Expires = DateTime.UtcNow.AddHours(1)
    });
}

Orphaned Blob Handling​

Orphaned blobs can occur when:

  • DynamoDB write fails after S3 upload (with NoCleanupStrategy)
  • Application crashes between S3 upload and DynamoDB write
  • Manual DynamoDB modifications

Prevention​

Use BestEffortCleanupStrategy to minimize orphans:

var options = new FluentDynamoDbOptions()
    .WithBlobStorage(blobProvider);  // Default strategy

Cleanup​

For existing orphans, consider:

  • S3 lifecycle policies to expire old objects
  • Periodic reconciliation jobs comparing DynamoDB references to S3 objects

See Also​