Ingesting Content into Cloud Service

Ingestion Process in the Cloud Acceleration Manager

Follow the steps below to ingest your migration set using the Cloud Acceleration Manager:

1. Go to Cloud Acceleration Manager. Click your project card and click the Content Transfer card. Navigate to Ingestion Jobs and click New Ingestion

   

2. Review the ingestion checklist and ensure that all the steps are completed. These steps are necessary to ensure a successful ingestion. Proceed to the Next step only if the checklist is completed.

   

3. Provide the required information to create an ingestion.

   • Migration Set: Select the migration set that contains the extracted data as the Source.
     • Migration Sets will expire after a prolonged period of inactivity, so it is expected that the ingestion occurs relatively soon after the extraction has been performed. Review Migration Set Expiry for details.
   TIP

   If the extraction is currently running, the dialog will indicate it. Once extraction has completed successfully, the ingestion will start automatically. If the extraction fails or is stopped, the ingestion job will be rescinded.

   • Destination: Select the destination environment. This environment is where the content of the migration set is ingested.

     • Ingestions do not support a Rapid Development Environment (RDE) destination, and they do not appear as a possible destination choice, even if the user has access to it.
     • While a migration set can be ingested into multiple destinations simultaneously, a destination can be the target of only one running or waiting ingestion at a time.

   • Tier: Select the tier. (Author/Publish).

     • If the source was Author, it is recommended to ingest it into the Author tier on the target. Similarly, if source was Publish, the target should be Publish as well.
   NOTE

   If the target tier is Author, the author instance is shut down during the length of the ingestion and becomes unavailable to users (for example, authors or anyone performing maintenance). The reason is to protect the system, and prevent any changes which could either be lost or cause an ingestion conflict. Ensure that your team is aware of this fact. Also note that the environment appears hibernated during the author ingestion.

   • Wipe: Choose the Wipe value
     • The Wipe option sets the destination’s starting point of the ingestion. If Wipe is enabled, the destination including all its content is reset to the version of AEM that is specified in Cloud Manager. If not enabled, the destination maintains its current content as the starting point.
     • Note that this option does NOT affect how the ingestion of content will be performed. Ingestion always uses a content replacement strategy and not a content merge strategy so, in both Wipe and Non-Wipe cases, the ingestion of a migration set will overwrite contents in the same path on the destination. For instance, if the migration set contains /content/page1 and the destination already contains /content/page1/product1, the ingestion will remove the entire page1 path and its subpages, including product1, and replace it with the content in the migration set. This means careful planning must be done when performing a Non-Wipe ingestion to a destination that contains any content that should be maintained.
   IMPORTANT

   If the setting Wipe is enabled for the ingestion, it resets the entire existing repository including the user permissions on the target Cloud Service instance. This resetting is true also for an admin user added to the administrators group and that user must be added to the administrators group again to start an ingestion.

   • Pre-Copy: Choose the Pre-copy value
     • You can run the optional pre-copy step to significantly speed up the ingestion. See Ingesting with AzCopy for more details.
     • If ingesting with pre-copy is used (for S3 or Azure Data Store), it is recommended to run Author ingestion first alone. Doing so speeds up the Publish ingestion when it is run later.
   IMPORTANT

   You can initiate an ingestion to the destination environment only if you belong to the local AEM administrators group on the destination Cloud Service author service. If you are unable to start an ingestion, see Unable to Start Ingestion for more details.

4. Click Ingest.

   

5. You can then monitor the ingestion from the Ingestion Jobs list view and use the ingestion’s action menu to view the durations and log as the ingestion progresses.

   

6. Click the (i) button in the row for more information about the ingestion job. You can see the duration of each step of the Ingestion when it is running or completed by clicking …, and then clicking View durations. The information from the extraction is also shown to realize what is being ingested.

   

Top Up Ingestion

The Content Transfer Tool has a feature that allows the extraction of differential content by performing a top-up of the migration set. This allows the migration set to be modified to include only the content that has changed since the previous extraction without having to extract all the content again.

To ingest differential content after some ingestions are complete, you must run a Top-Up Extraction, and then use the ingestion method with the Wipe option disabled. Be sure to read the Wipe explanation above to avoid losing content already on the destination.

Begin by creating an Ingestion Job and ensure that Wipe is disabled during the ingestion, as shown below:

Troubleshooting

CAM Unable to Retrieve the Migration Token

The automatic retrieval of the migration token may fail for different reasons, including you setting up an IP allow list via Cloud Manager on the target Cloud Service environment. In such scenarios, you see the following dialog box when you attempt to start an ingestion:

Retrieve the migration token manually by clicking the “Get token” link on the dialog box. Another tab is opened that displays the token. You can then copy the token and paste it into the Migration token input field. Now, you should be able to start the ingestion.

Unable to Start Ingestion

You can initiate an ingestion to the destination environment only if you belong to the local AEM administrators group on the destination Cloud Service author service. If you do not belong to the AEM administrators group, you see an error as shown below when you try to start an ingestion. You can either ask your administrator to add you to the local AEM administrators or ask for the token itself, which you can then paste into the Migration token input field.

Unable to reach migration service

After an ingestion is requested, a message like the following may be presented to the user: “The migration service on the destination environment is unreachable. If so, try again later or contact Adobe support.”

This message indicates that the Cloud Acceleration Manager was unable to reach the target environment’s migration service to start the ingestion. This situation can happen for various reasons.

• AEM as a Cloud Service maintains the environment state, and occasionally must restart the migration service for various normal reasons. If that service is restarting, it cannot be reached, but is available eventually.
• It is possible that another process is being run on the instance. For example, if AEM Version Updates is applying an update, the system may be busy and the migration service regularly unavailable. Once that process is done, the start of the ingestion can be attempted again.
• If an IP Allowlist has been applied through Cloud Manager, it blocks Cloud Acceleration Manager from reaching the migration service. An IP address cannot be added for ingestions because its address is dynamic. Currently, the only solution is to disable the IP allow list while the ingestion is running.
• There may be other reasons that need investigation. If the ingestion still continues to fail, contact Adobe Customer Care.

AEM Version Updates and Ingestions

AEM Version Updates are automatically applied to environments to keep them up to date with the most recent AEM as a Cloud Service version. If the update is triggered when an ingestion is performed, it can cause unpredictable results including the corruption of the environment.

If the “AEM Version Updates” is onboarded on the destination program, the ingestion process will attempt to disable its queue before it starts. When the ingestion is complete the version updater state will be returned to how it was before the ingestion(s) started.

If “AEM Version Updates” is active (that is, updates are running or are queued to be run), the ingestion will not begin and the user interface presents the following message. Once the updates are complete, the ingestion can be started. Cloud Manager can be used to see the current state of the pipelines of the program.

Top-up Ingestion Failure Due to Uniqueness Constraint Violation

A common cause of a Top-up Ingestion failure is a conflict in node ids. To identify this error, download the ingestion log using the Cloud Acceleration Manager UI and look for an entry like the following:

java.lang.RuntimeException: org.apache.jackrabbit.oak.api.CommitFailedException: OakConstraint0030: Uniqueness constraint violated property [jcr:uuid] having value a1a1a1a1-b2b2-c3c3-d4d4-e5e5e5e5e5e5: /some/path/jcr:content, /some/other/path/jcr:content

Each node in AEM must have a unique uuid. This error indicates that a node that is being ingested has the same uuid as one that exists elsewhere at a different path on the destination instance.
This situation can happen if a node is moved on the source between an extraction and a subsequent Top-Up Extraction.
It can also happen if a node on the destination is moved between an ingestion and a subsequent top-up ingestion.

This conflict must be resolved manually. Someone familiar with the content must decide which of the two nodes must be deleted, keeping in mind other content that references it. The solution may require that the top-up extraction is done again without the offending node.

Top-up Ingestion Failure Due to Unable to Delete Referenced Node

Another common cause of a Top-up Ingestion failure is a version conflict for a particular node on the destination instance. To identify this error, download the ingestion log using the Cloud Acceleration Manager UI and look for an entry like the following:

java.lang.RuntimeException: org.apache.jackrabbit.oak.api.CommitFailedException: OakIntegrity0001: Unable to delete referenced node: 8a2289f4-b904-4bd0-8410-15e41e0976a8

This can happen if a node on the destination is modified between an ingestion and a subsequent Non-Wipe ingestion such that a new version has been created. If the migration set was extracted with “include versions” enabled, a conflict may occur since the destination now has a more recent version that is being referenced by version history and other content. The ingestion process will be unable to delete the offending version node due to it being referenced.

The solution may require that the top-up extraction is done again without the offending node. Or, creating a small migration set of the offending node, but with “include versions” disabled.

Best practices indicate that if a Non-Wipe ingestion must be run using a migration set that includes versions (that is, extracted with “include versions”=true), it is crucial that content on the destination is modified as little as possible, until the migration journey is complete. Otherwise, these conflicts can occur.

Ingestion Rescinded

An ingestion that was created with a running extraction as its source migration set will wait patiently until that extraction succeeds, and at that point will start normally. If the extraction fails or is stopped, the ingestion and its indexing job will not begin but will be rescinded. In this case, check the extraction to determine why it failed, remedy the problem and start extracting again. Once the fixed extraction is running, a new ingestion can be scheduled.

What’s Next

When the ingestion has succeeded, AEM indexing will start automatically. See Indexing after Migrating Content for more information.

Once you have completed Ingesting Content into Cloud Service, you can view logs of each step (extraction and ingestion) and look for errors. See Viewing Logs for a Migration Set to learn more.