Looking under the hood - Cloud Manager 2022

Transcript

Okay, Remus, please feel free to start with your presentation. Thank you, Goran. Welcome everyone to the AMGEM session, Looking Under the Hood, Cloud Manager 2022. My name is Remus and I lead Cloud Manager Engineering. Today I will be co-presenting with Shankari who is my PM partner. Shankari? Okay, so as Goran mentioned, we also have Rahul Huda with us to answer any questions you may have during this session. Thank you for joining the session today and we hope that you and your loved ones are staying healthy. Without further ado, let’s get started. If you are not familiar with Adobe Cloud Manager, it is our purpose-built platform for our Adobe Experience Manager customers across both Adobe Managed Services and Adobe Experience Manager Cloud Service. It enables our customers to provision, configure and deploy to their Adobe Experience Manager environments in the cloud. There are four major themes tracked in 2022 in Cloud Manager. That’s reliability, constantly improving the success rate of our critical operations like executing a pipeline, making the pipelines run faster and as a result, enabling faster innovations for our customers, improving developer experience because we all know that happy developers build outstanding software in the long term and in the end, enabling customers to self-service an ever-increasing number of operations. The work on Cloud Manager started in 2017 and it was first launched at Adobe Summit in 2018. Since then, it had seen an explosive growth. Here you can see that pipeline executions saw roughly an year-over-year growth of more than 200% and the predictions for this year are that more than 1.2 million pipelines will be run in Cloud Manager. As the usage of Cloud Manager grows, running hundreds of thousands of pipelines puts a lot of stress on Cloud Manager and all the internal and external systems that are used. We recognized how critical Cloud Manager is to your business and want to ensure that we’re providing a constant level of service, so reliability was one of our top priorities. Therefore, Cloud Manager team has invested substantially to constantly improve the success rate of the pipeline executions up to 99% that translates in improvements to observability that allows us to detect and fix problems that may arise. Cloud Manager needs to adapt to different development styles and team structures, so pipelines are organized around three core types. There is the production pipeline, which builds code, runs code quality analysis, deploys to stage, tests the stage environment, and then deploys to production. The tests we use vary between managed services and cloud service. The managed services is focused on performance-based testing and cloud service is more functional testing and the experience of its step which evaluates delivered pages based on web best practices. There is a dev deploy pipeline, which does the same build and quality analysis and then deploys to a dev environment. And finally, there is the code quality only pipeline, which only does the first two steps. All three of these pipelines have a common set of elements. The UI is similar. Logs are presented in an identical fashion. The APIs are the same across pipelines. Events are emitted and webhooks enabled so that your external systems can respond to pipeline starting or ending. The build process and quality analysis are the same across these pipelines. So if a build works on dev, you are pretty guaranteed that it will work in production. What we have done is to layer a source code type on this existing model. So now for Adobe Experience Manager cloud service users, there are three types, three source code types that are supported. There is the full stack pipeline, which we have always supported and can handle anything that is deployed onto AEM or the Apache Dispatcher layer. We introduced the frontend source code type, which supports a new type of AEM site’s frontend project called a theme. And finally, there is the web tier source code type, which supports Apache and Dispatcher configurations. The key benefit we get out of this is that the build and deploy steps are different for each of these types and were optimized to only do what is needed for that corresponding project structure. To provide a bit more details, the full stack pipelines are limited to one per environment and their builds are based on Apache Maven. Frontend pipelines can support hundreds of distinct pipelines per environment, as AEM sites can support hundreds of themes per environment. The builds in these pipelines are based on NPM, adapted to the common workflows of frontend developers. Web tier pipelines are also limited to one per environment, as after all, there is only one web tier per environment. And the web tier pipelines, on the web tier pipelines, there is really no build script. There is a validation process that cloud manager will handle for you. And it should also note that if you use the web tier pipeline for a specific environment, the full stack pipeline that is deploying to the same exact environment will not deploy the web tier artifact. As I said before, these new pipelines are specific to AEM cloud service at present. We are, however, exploring options on how to make similar functionality available for Adobe managed services. Let’s talk a little bit on what improvements we have made in cloud manager pipelines this year. In 2021, we have seen a 20% average improvement in production pipeline performance. But let’s talk about what we have done to improve this existing metric further this year. In any given period, we found out that between 15 and 30% of the build steps were actually building the same git commit, the same code, the second time. That is a bit higher when we include Adobe internal usage. So we look at ways to optimize this and we introduced the build output reuse feature. The way this works is that when an execution starts, we capture the current git commit and then we check to see if we have stored artifacts for that git commit already. And let’s say that we haven’t. We run the build, check if it was successful, and if it was, we store the artifacts generated from that build, and then we continue with the execution from there. But if we do have existing artifacts, those are just copied and used as effective output of the build step. So we are not doing the build the second time. While this feature is enabled by default to all Adobe managed services and AM cloud service customers, there is a way to opt out of this if you wish to do so. Over the past several years, cloud manager has implemented some fairly aggressive retry policies to ensure reliable executions of pipeline. However, those retries have come with a performance cost. After all, we wait for something to fail before we retry it. So we looked at where we need to retry and the conditions under which those retries occur and identified that one of the most critical systems was the build service. The build service was created from day one with security in mind, and security was the top priority during the design and implementation phase. After all, we need to deal with code written by our customers that we both need to protect and also protect from. However, one of the challenges was that this build process was tightly coupled with the ACI, Azure Container Instances, and Azure Files. And this makes it harder to introduce additional execution environments like different Azure regions or different providers altogether. Design success rate is one of the key OKRs for cloud manager and one that we take very seriously. Any strategy that we adopt must put the security of our customers’ code first. Cloud manager’s new build service that was rolled out this year supports all Adobe managed services and AEM cloud service customers. This new multi-cloud build service has shown a whopping 99.99% success rate of the build operation. This new build service abstracts the build process and provides a standard contract for the build trigger and return of results. The pipeline triggers are built via a publish-subscribe-based fault-tolerant messaging system, and the build service is responsible for implementing the policies regarding where exactly the build is executed. It’s either Azure, AWS, or any other cloud provider in the future should we choose to do so. This insulates us from variances in regional and provider behavior without relying on retries. This new implementation also had a very welcomed side effect, which we anticipated but could not predict before the initial testing. We could not predict what would be the actual benefit, the extent of the benefit. We found out after the testing that the build step duration in the pipeline was reduced between a couple of minutes to sometimes more than 20 minutes. Obviously, this depends on the complexity of the code and the unit tests written by our customers. With that, I will give the microphone to Shankari. Thank you, Remus. I love how you’ve been able to share the exciting journey that Cloud Manager has taken in 2021 and the first half of 2022 with our audience here this morning. Now, I’d like to share some feature highlights, starting with 2021 and bringing you current to now. Let’s begin with features that directly address the pipeline duration centricity of our efforts over the past year. Re-execution of production deployment step. With this feature, a pipeline that has completed the production deploy step will display a UI affordance that you can see here to re-execute the production deploy step only. And this affordance is available regardless of the result of the step, be it successful, unsuccessful, canceled, as long as it is completed. Now, I’m going to switch to the live demo screen. Please let me know if you can see it. This is, I’m hoping you can see my Cloud Manager overview page here. Yes, we can. Great. Thank you for confirming. This should be a familiar page to most of you, the Cloud Manager overview page. Now, let’s navigate to the pipeline screen here. These are different pipeline types, full stack and web tier and front end that Remus had just talked about. The front end pipeline makes it possible to deploy the static parts of your website, CSS, JavaScript to a blog store and have them serve directly from the CDN in just minutes. The web tier config pipeline, on the other hand, makes it possible to very quickly deploy your Apache and dispatcher configuration directly for all your environments, again, in just a few minutes. So by enabling these diversity of pipeline types, different users or developers can deploy their pipeline very quickly. Now, let’s go over to the environment screen here. Here you can see a list of all of your cloud service environments. Let’s double click on this one. You can see all of the details here in case you’ve configured domain names or IP allow lists on this environment. Over here, you can see the preview service. Environments now have an additional preview service that allows you to review the content before it gets published to a larger audience, in most cases in the internet for public consumption. Let’s go over here to the configuration tab. From here, you can list, view or edit environment variables. You can add free form variables or secrets at the environment via the intuitive UI. Environment variables allow the behavior of custom code and application to vary based on context and environment. And these enable you to separate code from configuration and avoid including sensitive information in your version control. And by the way, these can be targeted to a specific service, author or publish. Now let’s go over to the restore content tab here. Restore content from Adobe backup is a feature that will allow you to self-service, do content restoration to return data that has been lost, damaged or accidentally deleted to its original condition. Cloud managers self-service restore process copies data from Adobe system backups and restores it to its original environment. Point in time restores are available for the last 24 hours and predetermined restore points are available for the last seven days. Now let’s look at New Relic APM and user management. We provide access to New Relic APM tool for all and it’s been added to all the environments for access to application performance metrics. And you can access your New Relic sub account from the cloud manager UI to quickly view and understand performance and application metrics for all applications or web transactions. Additionally, you can manage your users that you want to add or perhaps remove from the New Relic sub account directly via the UI.

Now let’s take a look at support for multiple repositories. Self-service creation and management of repositories via cloud manager UI means that, for example, front end and backend developers can leverage multiple repos, making it easier for different teams to coordinate and work independently and efficiently with clear ownership.

Next, let’s take a look at some of my favorite new updates in the experience to our customers on cloud manager. Onboarding customers who are getting ready to add in a program will now be able to submit their planned go live date in the add program wizard. A new go live widget in the UI provides guidance by tracking against this planned date. In this example, you can see June 29th, which is coming up in two weeks. And this widget not only tracks against this date, but provides in-product contextual links to useful resources in a timely manner. In this example, you can see that the link gives a checklist to go live for our customer to keep them on track and prepared. Next, let’s take a look at the new welcome banner that gives new onboarding users access to video tutorials at their fingertips. Whether it’s an overview to cloud service or how to add your program quickly or integrate with cloud managers Git repo or successfully configure and deploy pipelines right at your fingertips. Now, perhaps for more mature customers as they start adopting cloud manager, this banner displays a host of exciting progress metrics across all of your teams, across all of your programs so that you can easily know what the adoption and engagement across your teams is. Now, let’s take a look at the latest license reporting dashboard. A user in the business owner role can now access this new license dashboard on cloud manager landing page to view details of entitlements available in your tenant. We’ve started with sites, assets and forms will soon follow. Not only can you see solution entitlements available across all of your programs, as well as what is used and what is available by clicking the view usage details, you can see content requests, consumption metrics across all of your programs, trended by month for site solutions, which is where we have started. Additionally, cloud manager functionality can also be accessed programmatically via API. All of this is extensively documented for easy consumption for our users. This includes guides, tutorials and quick references. And now, let’s take a look at what’s coming soon in the second half of 2022. Content backflow. New automated content synchronization feature in between your AEM environments allows you to have your latest production content pushed down to your lower environments in a self-service manner directly through the cloud manager UI. Rapid development environments. This is going to be a groundbreaking feature allowing the creation of temporary AEM environments in the cloud in which light changes are possible. Create OSGI config settings, Java, JSP code so that you can instantly preview the result of any code change in your application in a well-controlled manner. And coming soon, secondary regions can be added to the published service on your cloud production environments. AEM cloud service relies on a list of primary data centers where content is stored and managed, also used as the origin of the CDN points of presence. We’ve added Singapore, Canada, Japan, Switzerland, India, and we’ll expand this list even further. These are all of the exciting things that you can look forward to coming up in the second half. And with that, I’d like to conclude my overview of the latest and greatest cloud manager features. And I’d like to open the session up for questions. Thank you very much, Shankari and Remus. So let’s check questions that have appeared. I’m looking at the Q&A tab. So let’s tackle the first question. When will hybrid AEM be on the roadmap? I don’t think I understand what hybrid AEM means. Yes, please, George, can you follow up on that? In the meantime, let’s have a look at the next question. What are these growth assumptions based on? And I think this is regarding one of your first slides, Remus, where you showed a graph when cloud manager started. That was in 2018 and showed a graph to 2022. But those were actual numbers, right? Pipeline usage numbers. That’s correct. So that graph, it’s based on the cloud manager observability that we have in place. And we are tracking all the pipelines that are executed through cloud manager. And the assumption on the 1.2 million number that we will hit by the end of this year is based on the number of pipelines that we have executed since the beginning of this year and a constant, not growth, but a constant rate of pipeline executions by the end of this year. So I haven’t even factored in additional growth in this year. So if we keep the current rate of today of running the pipelines in cloud manager, we will hit 1.2 million pipelines run in cloud manager by the end of this year. Thank you, Remus. In the meantime, we have elaboration on the previous question. Hybrid AEM is the ability to sync with a non-premise deployment of AEM. I’m not sure if that helps you. To sync content or code, probably both. I cannot answer this question right now about when this will be in the roadmap. I think we can note some questions and maybe answer to them later on. Goran, how is the process? Yes, exactly. Yes, we can follow up via contextual thread. Okay. All right, then let’s move on to the next question. How to revert to previous build on production using pipeline? Sorry, I… No problem. That’s the third question. How to revert to previous build on production using pipeline? To revert to the previous build… They would have to run the pipeline again. Run the pipeline again with a previous commit. Raul, you are also here. Do you have more context around this? Have we lost Raul? No, he’s here. Raul, you might have to unmute yourself. Raul, can you hear us? We don’t have an answer from him at the moment. Let’s follow up on that afterwards as well. Sorry about that. So the next question is, years ago you talked about working with Microsoft Azure. There was another question about Jira. Yes, please go ahead. But read it out. I’m not sure what you’re referring to. So it was moved to the answered questions. Okay. I don’t know why. I guess Raul just replied in this second. Oh, okay. I understand. He’s replying on… Okay. In the chat. So okay, the next question was, years ago you talked about working with Microsoft Azure. Can this be deployed in Azure web apps shelf? Do you plan to support Oracle cloud deployments? I assume by Oracle cloud deployments you mean AM running in Oracle cloud. Is that correct? If that’s correct, I do not have a right answer for you right now. I don’t know what are the plans for our group to support Oracle cloud as of this moment. Your assumption was correct by the way. Okay. All right. Then let’s look at the next question. What are my options to speed up Maven builds? Like using multi-threading or an external Maven build cache? I think there is documentation on helpx.adobe.com on how to improve the Maven builds or to optimize the AM code in order to run faster and compile faster through cloud manager.

Okay. Then next question. Does the build process work with non Adobe archetype Maven projects? We have existing projects in our customer archetype and how to migrate them to cloud manager. I think it should work with the, in the end the build service is running a Maven build process so it should work with other archetypes. All right. Let’s move to the next question. FE build pipelines. Can we configure separate pipeline for each FE modules? That should be possible. Yeah. Okay. Thank you. With the new introduction of multiple repo feature is still mono repo suggested or can we have multiple repo one for each LOB as per my current Jenkins pipeline? I think the whole goal there was to give the flexibility depending on the customer’s use case. So if they needed to have separate lines of businesses have streamlined and independent ability to work it really comes down to the governance and the requirements of that particular team and customer. Thank you. Shokari. And the next question, if we have technical questions with cloud manager set up, how do we approach Adobe through a ticket? Yes, that’s the, the current, uh, when recommended approach. Okay. Thank you. We answered all our questions for now. We’re going to wait some more to see if we are going to get more questions. So please post them into the Q and a tap. Okay. We have a next one. Is there a plan to make it possible to run it UI tests as part of a dev pipeline to serve as a gate for deployment? If so, when is that plan to arrive? This is to account account for a development lifecycle of IT UI tests. Yeah, we, we have a roadmap for this. I think as always, it’s a matter of prioritizing it. And I think in the meantime, customers are, I mean, the expectation is that you use the non production deployment pipeline and then run the tests in the production pipeline. And if needed, um, do not deploy the production, just go to stage. So we have a roadmap for this and it’s a matter of prioritization. Thank you. The next one is a comment might be related to a previous question. Currently we run a code deployment pipeline for full stack. It’s taking almost 40 minutes for one project. Yes, I, I’m aware of that. Improving speed for the cloud manager pipelines is one of our goals. And 40 minutes. Yeah, it’s, uh, um, something that happens. We have customers that run pipelines, uh, even for, for more minutes. It, I don’t know if it’s, uh, talking, um, about cloud service or, um, uh, many services, but 40 minutes, I would assume cloud service. It’s more likely, uh, usually on, on AMS it’s on AMS. Yeah. Usually on AMS it’s even longer than 40 minutes. Um, so I’m actually a bit curious on AMS between the stage deployment and the production deployment. There is a performance test that cloud manager is running for 30 minutes. So 40 minutes for, uh, a complete full stack pipeline seems a little bit low. Yes, it was mentioned that it was AMS. Yeah. And on the AMS cloud manager runs a performance test for 30 minutes. So just the performance test is 30 minutes fixed. All right. Thank you. The next question is, uh, please elaborate more on New Relic and is there any Splunk kind of log monitoring tool available on cloud manager? I don’t know what to elaborate more on New Relic. Um, Splunk available for customers. We don’t have that right now. The logs are available to download directly from cloud manager. That’s pretty much, uh, all that is offered right now in cloud manager. I don’t know if Splunk will help in, in more ways. Yeah, that’s accurate. Thank you. And yeah, New Relic is for, uh, the AM, uh, environments and, um, Thanks very much. Sorry? Oh, I just said thank you. Okay. All right. And let’s move on to the next question. As auto push updates done directly for production instance first, can it be in the reverse way? Like it will start from lower environments first, like dev stage and then production. So if any issue appears due to push update in the team can handle proactively on lower environments first. So today we make the update on the dev environment, something that the customer has to manually do test it out, feel comfortable with. Um, we do have a roadmap that we have prioritized, which is, um, which when completed will automatically first do an update on dev environment. And then a few days later will automatically then update the customer’s product stage. So this is something in the works. Um, and I’m glad to see this question cause it’s always validating, um, do that. We’re prioritizing the right things. Thank you. Um, there’s a comments, I’m not sure which question that was related to. So the developer developer needs to wait every time for dev deployment pipeline, which is longer duration and challenging. Understood. And I think that’s one of the reasons why the rapid development environment is something we’re working on to enable an H2, um, you know, different, there are different use cases. And we realized that one size is not going to fit all. So the, with the rapid development environment, the goal is that you can do multiple iterations of, um, you know, code changes and syncing it in with the rapid development environment in the cloud, and then seeing the impact of that and making changes rapidly multiple times without having to go through a deployment pipeline wait. And that’s exactly, hopefully that it will solve this problem. Thank you for elaborating Shankari. Next question is I am new to cloud manager. Is this for deploying CMS or for deploying external applications? It is for deploying specifically AM, not any CMS. Then next question in my currency ICD, we have some internal jar reference stored in the nexus repository, a private transaction jar. How do I get this moved for built, which happens in cloud manager since these jar are internal can’t be exposed to the internet? Yeah, this is a common scenario for a lot of, uh, of our customers and there are ways to connect from the build container to a private repository. However, on your side, you will need to put the IPs that we have for the build service into the allow list on your firewall. But we also support authentication from our build service to your private repository.

Thanks. Next question regarding the response to IT UI tests. Most customers require that all codes be tested before stage product deployment. The currency cloud manager architecture makes it impossible to meet customers requirements with respect to IT UI testing. Is there any other way to meet the requirement? I’m thinking that, um, maybe I’m not understanding very well the question, but, um, cloud manager provides web hooks that allows you to integrate with, uh, uh, the cloud manager pipelines and the run IT UI testing outside, uh, cloud manager. Thanks. Maybe to add here, I’m not sure whether here it mentioned that to test the code before stage, this is not supportive. What is to do the testing after the stage deployment, but before doing the production deployment and then it can be used via web hooks and adding a business approval manual step before going to production. So it stays there as long as needed in order to do the test.

Thank you, Ron. Got the next question with introduction of preview server. Is it possible for me to hold the workflow for production after author and preview and then do the deployment and publisher? Um, I, I don’t know that I need to follow up with, uh, with this question. Okay. Yeah, we can post it in, in the contextual front and we’ll follow up there. Uh, the next question is require access on Adobe cloud manager for deployment. Which team I have to contact for that? Uh, your own team manager role in cloud manager. You can do this. Your administrator, anyone who has access to admin console should give you a deployment manager role in cloud manager through admin console.

Thank you.

Um, next question is, could you please tell me more about New Relic APM tool integration? What metrics will it cover? Is it customizable? Some more knowledge on that. I will post a link to documentation for that. I think that would be the best thing. Yes, please. Thank you.

Then next question is dispatchers, CIS config variables and Chrome job expressions are directly configured in dispatcher server currently. Is there a way to configure those via cloud manager to avoid manual changes in dispatcher? Um, yes, but through Git. So we don’t have a specific UI to change CIS config variables or, uh, Chrome job expressions. So you put, um, these changes in Git and the web tier pipeline will deploy those changes on dispatcher. Thank you. Next question asks content sync is on the roadmap. Will it sync all content pages and assets from production instance to stage and dev as size of production environment can be in terabytes? Also, what will be the frequency of this content sync? I think you’re asking about the content sync. I think it’s a good question. Also, what will be the frequency of this content sync? I think you’re asking a lot of the questions that we are also currently working through, but at a high level, um, it will, uh, copy all content from production to a lower environment. Um, we are, one of our goals is to test the boundaries of what, you know, long running, uh, sync jobs, uh, what would the boundary of that be in terms of, you know, terabytes? Um, and as a function of that, we, we will, we may look into, uh, putting some limits on the number, the frequency of, you know, the, or the number of content sinks that we would allow, say, you know, in any given month or time period. So these are all things we’re working through. So, um, I think a little premature for us to have solid answers yet.

Thank you. Um, next question, any plans of improving the running instances, custom logs generation? We are always looking on, on ways to improve, uh, the transparency that we offer to our customers. So if there is something that is missing from those logs, just let us know, and we will see how can that be improved.

Yes, thanks.

Okay. I think there was another question in the general chat. Um, currently if I create a user in cloud manager, there is a decoupled user account created for Git in Adobe. But if, if I disable the user in Adobe IMS, um, but Git accounts still active for that user, which is a security vulnerability is getting addressed.

Um, that’s a good feedback. We usually, uh, allow this to be in control of, um, our customers so they can revoke access in cloud manager and, uh, revoke the, the, the tokens. But maybe there is a opportunity here to automate this and tie them together.

Thank you. Uh, we have one more question left. Uh, before, before leaving, I would like you to, um, fill out our concluding poll where, which is posted in the general chat and pinned to the general chat actually. Uh, also we have seven more minutes left. If you still have any questions, please post them into the Q and A tab. So the next question is it may be useful if there is an option to exclude certain nodes slash paths, which, which we need to exclude or merge during content sync. Yes, this is going to be, um, part of the feature, um, that we will enable and that’s great feedback. And again, uh, very validating that we’re going down the right path. Thank you. Is it possible to have logs of who launched the pipeline? Uh, this is something that we have a roadmap item for, uh, we, it’s a matter of, um, getting it prioritized as always. Thank you. The next question, get users access token are using Azure pet in the background, which expire after one year max. This isn’t documented anywhere by the way, and no notifications in advance as a comment. Thank you for this comment. Yep. Thank you.

Um, there’s another question. Can we add any step in pipeline like, uh, at regression suit before deployment? I would say same as what Raul and Remus explained, use web hooks to do that potentially.

Thank you. And there’s another question. Is there any customers? Slack matter most channel notification available for pipeline running jobs status.

Um, no, but I think that can be implementing implemented very easily using the web hooks we provide. All right. Thank you. And the last question for now, Adobe cloud manager is using which cloud provider as your AWS or Google.

Does it matter? Does it matter? We run these pipelines and install, uh, customer code on environment. If you are interested on where am it’s running, that’s a different story. Um, George is mentioning here patient. I just shown, uh, that we are not only Asia in the presentation before that.

All right. Great. Thanks for answering all the questions. Remus and Shankari and Raul. Uh, to the audience, any other last questions? We have a couple of more minutes left.

We’re going to wait a short moment to check if more questions appear. Otherwise we will close this session.

There’s the question whom to reach to get the webhook details.

We have documentation on this. I can share it with on this. Thank you.

And is there any plan to upload artifact directly instead of pushing code? I mean, build code in my local org and just push binary package to cloud manager.

No, there is no option right now provided for that.

Okay. Thank you.

All right. We would have time for one more question.

All right. It doesn’t seem to be the case. Yeah. If you still should have questions or, uh, after the session, you, you realize you do have questions, then just go to our contextual threads. There you can interact with us and with the community. Uh, our recording of the session will be made available also on this contextual thread as well as in the sessions archive on adobe.com slash go slash gems. I would like to thank you very much for joining and for your attention, for giving us feedback and to post all these questions, which are also valuable to us. Uh, and, uh, with that, I wish everyone a great day or evening and goodbye. Thanks a lot.

Thank you. Have a great day. Thanks. Okay.