AEM as a Cloud Service: Recommended strategy and limits for Bulk Programmatic Asset Upload (~50,000 assets)
This article provides guidance on best practices, recommended limits, and configuration for bulk programmatic asset uploads to AEM as a Cloud Service using direct binary upload.
Description description
Environment
Adobe Experience Manager as a Cloud Service (AEMaaCS) (all versions)
Issue/Symptoms
You need to ingest approximately 50,000 assets into the AEM as a Cloud Service DAM from an external system using the direct binary upload approach with the @adobe/aem-upload Node.js library.
You seek confirmation on the recommended strategy, supported throughput, batch size, concurrency, and any official limits to avoid overloading the Asset Compute Service or triggering throttling. No error messages have been reported yet, but you want to validate the approach before production ingestion.
- Bulk upload volume: 50,000 assets
- Approach: Direct binary upload via presigned URLs (
@adobe/aem-upload) - Batch pattern: Batch Upload with retry logic
- Batch size: Documentation suggests 10–50 in production
- Concurrency:
withMaxConcurrent()method available - Authentication: Service credentials (OAuth S2S / JWT)
Resolution resolution
Follow the steps below to resolve the issue:
-
Use Direct Binary Upload
- Follow the Programmatic asset upload tutorial and Asset Upload HTTP API reference.
- The direct binary upload method is recommended for large-scale ingestion, as it uploads files directly to cloud storage and minimizes load on the AEM JVM.
-
Configure Batch Size and Concurrency
-
Use the Batch Upload pattern as described in the documentation.
-
Recommended starting configuration:
batchSize: 20–50 files per batchmaxConcurrent: 5–8 concurrent uploads per client process
-
Adjust these values based on pilot results and observed system behavior.
-
-
Monitor Throughput and Processing
- There’s no strict hard limit on requests per second or assets per hour for the upload API itself.
- Actual throughput is governed by network bandwidth and client concurrency.
- Post-upload processing (thumbnails, metadata, Dynamic Media, etc.) is handled by Asset Compute microservices. Typical processing rates are 5,000–9,000 assets per hour.
- For 50,000 assets, processing should complete within a few to several hours under standard conditions.
-
Pilot and Monitor
- Run a pilot with 1,000–2,000 assets to validate configuration and monitor for errors such as timeouts, HTTP 413 (Payload Too Large), or HTTP 429 (Too Many Requests).
- Monitor processing times and *Processing…*banners in AEM.
- Adjust batch size and concurrency as needed to optimize throughput without triggering throttling.
-
Authentication and Permissions
- Use service credentials (OAuth S2S / JWT) with an account that has the required upload permissions.
-
Schedule Uploads Appropriately
- No special environment scale-up is required for this volume.
- Consider running uploads during off-peak hours to maximize processing throughput.
Cause
The direct binary upload approach is designed for high-volume ingestion and leverages cloud storage and microservices for scalability. Throughput is primarily limited by post-upload processing capacity, not the upload API itself.