Freeze, Restore, Innovate: Hands-On with Rapid Development Environment Snapshots for Cloud Development

Transcript

Welcome to the free 3-Store Innovate, hands-on with RDSnapshots session here on demand for DevLife.

What do we talk about here today? We basically go through how you can implement, run and test code in an AEM on the cloud in a rapid, fast manner. And you can also replay at any time using snapshots.

I’m your host, Remo Lichty. I’m a software engineer in the Foundation Core Cloud Service team. And I’ll talk you through these demos today.

Today we go through the basics of RDS.

So why should you use an RD? How to use it? If you’re already familiar with that from previous sessions, feel free to skip over to snapshot indirectly, where we basically will talk about what do snapshots provide? Why is it a thing that will speed you up even more? And then we go over to some demos.

And I’ll show demos for a basic RD interaction and also deep dive on snapshotting in the next few slides.

So what are the basics of an RD? So RD’s allow you to hot deploy code and content and give you immediate fast feedback. Remember, if you run a pipeline with a YAM on the cloud, it roughly takes 20 minutes until the pipeline runs through and everything is deployed on your dev environment.

With RD’s, that’s different. So you basically can run quick experiments. You just install a couple of files onto an RD. So the RD means rapid development environment.

Basically it gives you the possibilities to analyze issues on the cloud because they are very similar to regular AEM cloud instances. For example, you have add-ons available like forms and assets and others, and you have services integrated with the cloud. So it’s different than when using QuickStart locally and gives you a way faster approach of working directly in the cloud.

There’s compromise, however, so we don’t have high availability of RD’s because it’s a development environment. So there’s no multiple author or publish instances. There’s just one publisher, one author instance. So that’s the basic difference there. And on the next slide, I will go a little bit deeper in the workflow of how you interact with an RD.

So how do you interact with an RD? So basically you have a CLI, so a command line interface, user terminal, for example, where you can use the AIO tooling. So Adobe IO. We have a RD plugin for the AIO, and that one offers different commands to interact with the RD. I’ll come to the details of those commands later on. But basically if you, for example, install files, we access your file storage locally in step number one, and then the AIO command uploads that files in step number two to a file share where the RD instance then goes and grabs it and installs the files into the author and the publish pod by default. You could however also target one or the other if you’re just interested in working on offer and you don’t care about publisher for now.

But usually default wise, we just install the files in both instances and that’s basically how the entire solution works. So it’s pretty straightforward.

So for you as a developer, you just have a tool to install on your computer or the AIO.

That’s usually like three commands to do. There’s a documentation about that. I’ll share that on the last slide with all the details. So you install Node.js, for example, and then you install the AIO basic tooling. And on top of that, you install the RD plugin and then you’re ready to go.

So what are the basics of the commands to interact with the RD? First of all, all the commands that we provide come with a dash dash help that gives you a deep insight on how to use it, what options are available. Sometimes even provide you some examples on how to call the commands. As well as some of the commands come with dash dash JSON, which allows you to use the tooling programmatically. So for example, if you run a CI CD environment like Jenkins, you could use the AIO to, for example, install files in a Jenkins job and then run unit tests on that. And when you’re done, reset the RD afterwards to get it back to a vanilla state cycle storage.

So make sure always to check the help. The help will also mention if there’s a JSON option and yeah, let’s dive a little bit into the different commands. It’s not all of the commands that we have, but those are the basic ones that you need to know. So first of all, it’s the setup command. That one gives you a simple way to set up the connection to your RD with an interactive user guided flow. So it will allow you to store things locally, globally, depending on your needs. I’ll show more about that in the demo later on.

It gives you the possibility to log in to your organization and choose your RD from your organization without knowing the program ID or the environment ID. So it’s really an easy way to set up. Then the status command, that one actually just goes on to the RD and grabs all the installed files and lists them to you with more or less details. If you don’t specify the dash dash JSON option, for example, you just see like the configs and the bundles which are installed with their IDs, that’s it. If you choose to show the dash dash JSON, you get way more information about different metadata, which is stored alongside with it. So that’s always good to know that with the dash dash JSON, you will get way more information out of the status. Then history. Yeah, obviously if you install delete files, you leave a trace of a history. The history basically just shows you what you have done, what files have you installed, what have you removed and gives you an idea of what was deployed alongside with error messages, for example.

There could be the scenario where you install a file and then it has not all dependency resolved. So it will mention like, yeah, it has been staged, but you need to upload a dependency and once the dependency is satisfied, then it will install eventually. So those are the things that you also see in the history.

It’s always good to check back the history in case you installed something and it didn’t work out the way you intended it to do. Which brings us back to the install command. So basically within the install command, you install files. You can install configurations, bundles, front-end, things.

It basically gives you all possibilities that you can do of installation code and content onto an RD. It allows you to force install and overwrite certain other files.

And obviously once you installed, you maybe wanna remove some of the files. So that’s where we have the delete command for you, where you can specify config IDs or bundle IDs to be removed. If you’re unsure what is the config and bundle ID of something you wanna remove, remember it as status dash dash JSON will give you those. So always work together with that. Then we do have a relatively new command for logs. So this gives you the possibility to tail logs on your RD. So basically really hop into the RD instance of or publish or both and get a flow of the logs going to specify what are you interested in looking at, like a little grip. So for example, you would just wanna see debug messages for com.adobe and you can do that. We have a demo later on that shows you how to do this. It really helps you to look inside the RD with very basic tooling only. So you just need a terminal and the IEO and the plugin and you’re all set. And last but not least, we have to reset command. So somewhere there’s for sure the time where you have worked a lot on the RD and it has blown up with content or you installed stuff which is broken for some reason because of a bug and then the RD does not recover or you need to basically get a new AM version as well to upgrade your RD to the latest AM version. That’s where the reset command comes in handy. So that one basically just wipes everything from the RD, gives you a new RD and in the same go updates to the latest AM version.

Yep, so I think that’s it. Let’s jump into the demo later on.

The basics of the interaction with an RD is going through the AO commands. So we have to plug in four RDS in the AO space. And if you’re unfamiliar with the commands that it comes with, just look at the help. The help is available for topics like RD or also for individual commands.

The best basically now is to first establish a connection to the RD and we do this by calling the setup command.

The setup command basically gives you an interactive way of using the configuration of AO. So first of all, it asks you if you wanna store the information, the logging information locally. If you’re working with more than one RD and you tend to keep your connection information in the .AIO file within your project, then you should use the local storage.

If you just use a single RD and basically just use that RD for development, no matter what project you’re working on, you basically can use the global storage. So that would be a no here if you do that. We now choose to store the information locally.

Then the next thing that you could ask is if you wanna get desktop notifications shown for long running processes like snapshotting or resetting or restarting the RD, that’s always a good thing to do. So we do that. And basically now chooses to show all the programs of your organization. I’m here currently on a staging environment for development, the RDS.

So here I can basically go on and say, okay, I wanna see the RD development space. I can filter by typing and I can navigate with the down and up arrow keys.

So I’m using that program. I hit enter and basically it lists me all the RDS together with their state. And I’m basically using my own Europe based RD for the demo here. So I can basically just select this one and my connection is now basically configured. To validate the connection, it’s probably easiest to do the status command real quick to see at the first glance if we have something on there. So we don’t, the connection was working fine. And that basically is how you set up the RD.

In this little demo, we’re taking a look at how to install files to the RD, see what’s installed on there, how it has changed over time, and how we get rid of them again afterwards. First of all, we’re taking a look at the status of the RD and basically showing what is also installed there. So we see that so need already, there’s not been installed on it. Let’s change that. We’re going to install a little demo that I have prepared.

So we’re gonna upload that zip file.

It contains a few little servlets and so it should be done fairly quickly. It’s just 1.3 megabytes. So basically we have uploaded that one to the RD and now it’s basically installing the files and applying them. So while we wait here for it, and we see that took a couple of seconds and that’s actually the number six deployment on that RD.

So we can go on and check now again with the status, what has been applied. We see that we do have some bundles on offer and publish and some configurations, which are there as well.

Also, good to know is the minus, not the dash dash, JSON option, which basically gives you an insight with more details on what’s installed. In example, the bundle names or IDs of the conflicts that you later on gonna use when you choose to remove it, then again from the RD.

Also, we can go in and check the history. So let’s see what has changed.

We see that was the deployment that we just did and why is this number six, but only the first message here. That’s because I had used it already before and was installing things and then I chose to reset it. However, the number of deployments, that’s an account which is persisted. So now we see that this is the case here.

So yeah, what else can we do now? We can basically go on and delete some of the files.

In example, I choose to delete that config.

So it’s deleting it. And then there’s also the other config or the other bundle that we can choose to delete. So we use the symbolic name for that later on. So we do a, waiting here until it finishes completely.

All right, we’re also going to delete that bundle.

And once this is done, we can see already now we had number six, the deployment of the demo project and deleting was number seven. And then also the deleting on the publish. So seven was on author and number eight was on publish. And now we are deleting it number nine from author while we wait for number 10 to have been deleted from publish. So this is done. Let me clear the screen real quick.

In the history, we now should see six to 10. So that’s what we just have done. Like number 10 on top, most recent one.

And status with JSON for all the details should basically show that there’s nothing installed anymore. So in a very quick overview, that’s how you install, check for installed stuff with history and status and how you delete later on what you have installed previously.

In this quick demo, we’re looking at the logs functionality of an RDE. As you know, with YAM on cloud services, it can be hard sometimes to retrieve the logs quickly. And within RDEs, we made that very simple. So you basically just type in the RDE logs command and let’s take a look at the help first. That gives you a brief idea of what’s possible. First of all, you can see the different log levels that you can take logs for. Plus you can have something like highlighting that you can substring a log line to give a different color.

So yeah, let’s jump right in. Let’s do a minus T for Comadobi to see what we are logging there for debugging information.

So we see, for example, there’s stuff regards, slim discovery or other stuff like page producer strategy. Maybe you’re interested in that one. Let’s highlight this one a little bit.

So we have a different color and we can immediately see when this one pops up. Okay, there was some.

So that’s basically how you can really quickly get insights into the logs. Also, what’s interesting there is you can target to publish or the author pod individually by using the .s or dash, dash target option, where you just specify author or publish. The default is author, so you only get those logs, but obviously you can also get others. If you don’t specify anything, we are basically just tailing the logs, which are on info and for none of the packages, so you just get all the info logs. For example, if you are opening pages on your RDE and browse stuff, then you will basically see all the requests and the responses there. So that’s interesting to see. And if you need more information, always feel free to do maybe something like this com.adobead we have done before or minus E or let’s say, we don’t have errors too much there but maybe something on day as well and different log levels, they also have different coloring. So for now we only see debug ones and that’s because I don’t have really something running on the com.day space for now, but basically you can do all the different log levels and in the same time.

So yeah, I hope that gave you a quick overview about what’s possible and it’s your insights into your code, which is running on the RDE and yeah, have fun.

So after looking at the basics of the RDE, we now deep dive into snapshotting. What does snapshotting give you? It gives you the possibility to capture the state of your RDE and put it aside and later on use it again. Once you have done your work and you wanna put your RDE back into a state where you can have a good headstart. So that’s useful for example, if you have testing to do and you need to revert your RDE to the initial state with all the test data and then do another test. So that’s where snapshots come in handy or for certain configurations that you need to do over and over again, whenever you reset an RDE. So it’s really handy to have those available. So a snapshot contains the deployed code and the content that you have installed previously. So everything that you have used the AMRDE install command for is basically going into a snapshot.

Also, if you would delete some of the files and take a snapshot afterwards, the files which are currently installed on the RDE, make it into the snapshot and that gives you the possibility to store that away and use it for in a later time point. You can restore snapshots to your RDE. So this basically wipes all content and deployment on your RDE and overwrites it with the content and code from your snapshots.

So it basically starts up the RDE where the snapshot has been taken. You can restore as many times as you want and you can also delete snapshots that are no longer needed. So it’s always good to maintain the list of snapshots to not overgrow so that you have only the snapshots at hand that you actually need.

What are the snapshot commands that we provide? First of all, we have a topic called snapshot. So AIO, AMRDE snapshot. This one lists all available snapshots. And if you execute the dash dash help parameter as well on the snapshot command, it will list you all available commands and how to use them. And there’s also detailed information on each of those commands with the help function. So make sure you call that once to get an idea of what’s possible. But the basics are create, restore, delete and undelete. So let’s talk through that one real quick. So create obviously create snapshot. To create a snapshot, you need to provide a name. You can also provide a description if you like to give some more details of what’s the content of the snapshot. With the restore command, you can restore a snapshot and restore the content and deployments back into your RDE.

And then you can also manage the snapshots with delete and undelete. So you can delete the snapshot and that one basically changes the state of the snapshot to be deleted and not available anymore. So only available snapshots can be restored but deleted snapshots cannot. And the reason why we have that state is now imagine you have spent a couple of hours to install all the files and everything to set up in your RDE the way you need it for testing or whatever. You make a snapshot of it. And then later on you by mistake delete that snapshot because you had a typo or you had the wrong name in there and boom, your snapshot and all your work is gone. So that would be terrible. And that’s why we have the delete snapshot command which will change the state from available to delete it. So you can always go back and undelete it. So with the next command, you can recover a snapshot that was accidentally deleted and make it available again. So snapshots that are in a deleted state will remain for seven days and only after seven days they will be wiped. So this gives you an extra safety net to not delete your work by accident or that some other worker coworker in your company have deleted the wrong snapshot due to a misunderstanding and you can always go back and make it available again. So you can keep those for seven days. If you’re really sure, obviously the delete command gives you some more options to first delete a snapshot right away and wipe it and not just keep it for seven days. For example, if you need some more space because the number of snapshots is limited and you can only make a couple of them. We do this to prevent that we grow a huge disk volume on the cloud. Now just imagine if you have 10 gigabytes of content on your RDE and you have a hundred snapshots. I mean, that would be huge. So that’s why we have basically just limited the number of snapshots for now. And we give you the option with the delete command to wipe them instantly or to keep them retained for a week.

In the demo, you’ll see all that. So we go through snapshot listing, creation, restoration and managing with delete and undelete later on.

So let’s deep dive into snapshotting.

First of all, you need to learn the commands that come with it.

So let’s see what’s there. We have a topic for snapshotting called snapshot on the RDE plugin. It basically shows you the different commands. So you can create snapshots, you can restore them to your RDE you can delete and undelete them. The delete and undelete will be covered in a later demo about managing snapshots while the restore will be also in the next demo actually. So for now we deep dive into creating a snapshot.

So let’s start right away.

So yeah, the snapshot itself lists you all the snapshots that you have in your system. Apparently I don’t have one anymore.

So I need to create one. And how do I do that? I basically say, create me my first snapshot.

I can specify a description.

This is my initial snapshot.

So yeah, let’s fire that one away.

This basically is put in four different steps. So first of all, we request that we want to create a snapshot to the backend. This is usually pretty fast. So the backend confirmed that we wanna create a snapshot and now we’re waiting for the backend to basically pick up that job to go on and create a snapshot that’s usually within a minute or so.

And once this has started, RDE is being locked and the snapshot is being created. So the RDE needs to be locked because we cannot create a snapshot while you are still able to install more files, because then we would only have partially those files in and that’s for sure nothing you want to have. So that’s why we are basically locking the RDE.

So now you see the backend has picked up that snapshot. Now it’s locking the RDE right away and avoiding or preventing that somebody resets that RDE or installs new files. You could still though work on the RDE in a way that’s for example, you query for status, you can still query for logs, stuff like that, but you cannot change the RDE basically.

So creating a snapshot usually takes a couple of minutes, depending on the size of your RDE.

And as you have seen what we installed on that RDE in the previous demo was just like a bundle and the servlet, so not really much. The snapshot should be, I guess, roughly around 40 megabytes.

So all in all, this takes some time. You see now it took a minute to create a snapshot, but depending on your content, this can take an hour as well. I mean, if you have like 10 gigabytes or something of assets stored on your RDE for some reason, obviously copying that data takes time. So it really depends on your content.

There’s a bit of ramp up time in creating snapshots. So even if I would have now probably a couple of hundred megabytes, it would not take much longer because all the backend needs to spin up and the files need to be created and stuff like that. The RDE needs to be locked and unlocked again. And that usually takes the same amount and no matter the size of your RDE, the locking and unlocking for sure.

So yeah, what we see here is like the minimum time it takes to create a snapshot.

And that’s basically what is happening here. So be prepared. If you take snapshots that are bigger, they obviously take longer. It’s always a good idea to have as little data in your rapid development environment because otherwise it’s not being rapid anymore, right? If you have many, many gigabytes, it will just slow down in snapshotting. And it’s always good to limit yourself. So you see now the total time to create a snapshot was around three minutes. And that’s pretty decent, I would say. And if we now wanna see what is the snapshot size, for example, we go in and call the snapshot function again. And we see my first snapshot was created. We have not used it yet. We have not restored that to the RDE. It’s 30 megabytes. It’s available, which means you can use it to restore and it has not yet been used. So that’s why those two fields are empty, basically.

So yeah, that’s how you create a snapshot, as simple as that. And you can create more than one snapshot, obviously. Maybe let’s take another one.

Just so you see how this works.

This will take another three minutes, I guess.

And why I’m showing you this is that we can see then how the list grows and as well, that we can basically later on use that snapshot as well in the upcoming demo for restoring so that we have some changes there. I’ll stop that right now for here. Because you’ve seen how this is done, but you will see that a snapshot has been created in the next demo.

Let’s take a look how we can use and restore snapshots. First of all, let’s take a look what we have in our RDE.

And since that RDE should be empty, we should not really see much here. Yeah, exactly. So that’s an empty RDE. And now let’s take a look what snapshots that we have. We created one before, so there should be one. And that one is actually already being used once because I tested it before. So let’s use it again. And let’s dive right into that one. So we say we wanna restore snapshot first. And this is basically doing the same four steps than the creation that we had seen before. So first of all, which is pretty fast, is requesting the backend to restore the snapshot. And now we’re waiting for the backend to pick up to request a job to restore the snapshot. That usually takes a couple of seconds until the backend has been up the job.

So I can see in my other logs in another window that this one is actually now ramping up. So within the next couple of seconds, we should see that it actually has done that and jumps to the third step, which is restoring the snapshot itself.

So this is basically taking a couple of minutes. And then the major difference here compared to the CreateSnapshot is that we changed the underlying data. So we have to restore the RDE when we have restored the snapshot.

And when creating snapshot, we just had to unlock the RDE, right? We did not change the RDE. We just took the data from it, copied over, but we did not change the RDE. And here with restoring, we obviously do change the RDE. And that means we have to restart it. That usually takes a couple of minutes longer.

So yeah, let’s just wait for both to happen. I’ll speed up the section in between real quick to not bore you with waiting time.

So we see the restart took almost four minutes and in total, it took like a little bit more than five minutes to restore the entire snapshot.

We can also now check if we have stuff available. So if we go with the JSON status again, we should see that now we have the demo bundle and the demo servlet there as well. We can also curl that servlet and we get a response. So it’s obviously there, it’s working.

And what else can we check? We can go and take a look at the snapshot list again and the usage should be increased from one to two now. Exactly, we see that it has been used last time a couple of minutes ago. So that all works and gives us an indication by the way, you can also use the help function to find a sorting mechanism from the list.

So here is the flag to sort.

If you have many snapshots, then you can sort by number of usage, for example, and that will help you to understand how frequent your snapshots are being used.

Obviously, yes, I agree. Your next question probably would be installing the file took a minute and now restoring the snapshot actually took five minutes, that’s way longer. And the snapshot is only 25 megabytes. Yes, that’s true. The major part of the time of the entire restoration of the snapshot is basically not copying those 25 megabytes or installing the files. It’s spinning up all the backend infrastructure to basically acquire the snapshot file and move it over and then start restoring it. And then as well as restarting the RDE. I agree for such a small two file deployment, that’s probably not really a use case, but in your case, you usually have a lot of data in there, a lot of content, many assets, and a lot of code or configuration that you need. And all this takes time to set up and when it’s really, really worthwhile working with snapshots. Here, I just limited it to the very small bare bone so that you understand how it works. And I guess you got the idea. Now in the next demo, we’re gonna take a look how we manage with delete and undelete.

In this demo, I’m going to show you how to manage snapshots. First of all, you need to take a look what snapshots are there. And so we do this by typing a snapshot command. We will just start to list all the snapshots.

What do we see here? First of all, we see a name and a description. I have created a bunch of snapshots here to fill the list so I can show you something else as well later on. We see the size. So how big is the snapshot? When was it created? When was it last used to restore? So that is for now empty since I have not performed a restore yet. And you can also see there’s a state of the snapshot. It’s either available. That means you can use it to restore it to your already or it’s in state deleted which means you have previously deleted that snapshot. We keep snapshots that are deleted for seven days. So we retain them for a week. In case you accidentally deleted the wrong snapshot you can undelete that snapshot in case you still need that one. So yeah, let’s play around with the states a little bit. First of all, I would undelete my first snapshot.

So let’s do that. And then you basically see that if I list it up again that the state has changed from deleted to available which means I can now use that one as well.

Yeah, same thing for deleting.

So let me delete the first one. Obviously this one is going to change the state from available to deleted if we call or list the snapshots again.

So that’s basically all you can do here. Some more information on the delete command basically is that you can force delete a snapshot which will skip the retention time of seven days. So it will be wiped from disk right away and you can also delete all if you want to for some reason.

So yeah, there’s also a limitation of how many snapshots you can take. For example, if I try to do another one and I’m trying to create another one, it will fail because I have already too many snapshots. And so I basically need to get rid of one before and I can do so by deleting all.

Oh yeah, you still need to specify one snapshot as a name because the name is always required, but it’s going through and now deleted all my snapshots which means I can go on and list the snapshots and basically I don’t have any snapshots anymore. So they got wiped.

So we’re on the last slide. I hope you had a good introduction to the RDE basics and then further on on the deep dive on snapshotting with all those demos.

I wanna show you some limits and the forecast about snapshotting. So right now snapshots are tied to your RDE. So you can only restore a snapshot on that RDE where you actually created it from.

Yeah, we know that’s not really handy, right? So if you have a bunch of RDEs, you basically wanna be able to create a snapshot on one RDE and then restore it to a bunch of different RDEs. And we definitely understand that. We will be working on this. It’s on the roadmap so that we can basically share all snapshots of RDEs within the program so that you can use that and share all the setup, all the content and code that you need, for example, for a CI CD environment, to spread that all over your RDEs if you need to. We maybe also will think about keeping some snapshots private so that you can mark them to be tied to a single RDE and not being usable for another one. But it really depends on the use cases that you guys have. And that’s why we would be really interesting to hear from you about your use cases.

So snapshots are not yet GA, so it’s not generally available. You can, however, contact your TAM, your technical account manager, and ask to get access to the early access program. So you gotta check some boxes and some paperwork, shouldn’t be too much effort to do. And then we can enable your program on snapshotting and also work with you together on the feature set, for example, about sharing and maybe keeping some snapshots private or not. So we will be really keen to know and understand your use case. So please, if you’re hyped as much as we are about snapshotting, then please reach out to your TAM and get this started, so we can make it even better for all of us. So I hope this was a good introduction for you. And if you have questions, there is a Discord channel about RDEs in the AEM server of Discord that you probably have heard of with the H3L services or others AEM topics there. So watch out there for the Discord channel for RDEs. On the last slide, on the very last one, I’ll show a couple of links and that one is a little bit there. So yeah, reach out to us. We’re happy to talk to you and yeah, have a great rest of the life.