Best practices for high-volume API imports in AEM as a Cloud Service
This article explains how to optimize large dataset imports into Adobe Experience Manager as a Cloud Service (AEMaaCS), especially when direct, high-concurrency API calls fail due to platform constraints.
Description description
Environment
Adobe Experience Manager as a Cloud Service (AEMaaCS) (all versions)
Symptoms
When integrating external data sources with AEM as a Cloud Service, you might want to import large volumes of content, such as thousands of articles. Sending a high number of concurrent API requests (For example: 50 or more at once) result in failures because AEMaaCS enforces strict timeout limits for outgoing connections. Integration flows that worked on AEM 6.5 won’t function the same way on AEMaaCS due to these architectural differences. Adapting integration strategies to the cloud platform’s constraints is essential for reliable and scalable imports.
Resolution resolution
Follow the steps below to resolve the issue:
- Avoid sending a large number of individual API requests concurrently to AEM as a Cloud Service, as this exceeds platform timeout limits and causes failures.
- Aggregate records into batch files (For example: group every 50 records into a single file) using your middleware or integration layer.
- Transfer these batch files to a location accessible by AEM as a Cloud Service. Ensure the transfer method and storage location are compatible with AEMaaCS connectivity and security requirements.
- Develop or configure a service or scheduled job within AEM to process these batch files asynchronously. The service should extract and import records at a rate that doesn’t exceed the platform’s connection and read timeout limits (10 seconds for connection establishment, 60 seconds for read operations).
- Monitor the import process to ensure that batch processing completes successfully and without timeouts or failures.
- Adjust the batch size and processing frequency, as needed, to optimize throughput while staying within platform constraints.