Custom Functional Tests for safer CI/CD pipelines

Transcript

So I guess we can get started. Hello and thank you very much. I’m Andres Bot. I’m a cloud software reliability engineer. I’m currently working on AM as a cloud service. And in today’s session, I’m going to talk about customer functional tests and how you can leverage this tool in your cloud CI-CD pipeline in order to have more confidence on production deployments. So as you might know, among many of the benefits that cloud service has is a quick time to market so you can quickly deploy new changes into production. But with this velocity, we also have the challenge to not break the actual customer experience. We can no longer rely on weeks of manual testing. So for this reason, cloud manager or cloud service has many quality gates that verify the different steps of the product and your application in order to make sure that you are confident on the deployment to production. The custom functional test step is your way that the application has where you can add an additional level of quality gate that will be executed as part of this deployment and verify your application and make sure that you don’t break the application when deploying to production. This applies both to your own application code and to the product push updates. So whenever a new version is pushed to production, your custom functional tests will also be executed as part of this deployment. Before we dive into the custom functional tests, I would like to talk a little bit about the product functional tests since they share the same technologies as the customer functional tests and they run on the same platform in the same methodology. This is a set of tests written and maintained by Adobe and their main intent is to prevent the application code to break the core functionality of AM. So let’s say for example, if you have some code in your application that interferes with replication, there is a specific replication test in the product test that will prevent this code to be deployed into production. These tests are open sourced and built on best practices and you can find them on the GitHub repository called AM test samples. I have a link in the reference later on. So now for the custom functional tests, this is the part of your application that is intended to test your application, the interaction of your application with the whole AM stack. So that means in this case, it’s going to be on author. If we’re talking on author instances, we are on load balance instances. If we’re talking on a dispatcher, remember there is always a publisher in front of the dispatcher and all in all, we’re talking about your code deployed in AM and on the cloud service, the whole stack. These tests are executed as part of the production deployment pipeline and they block the production deployment if they don’t pass, but luckily you can also run them locally. So these tests are written in Java. They use the AM testing clients and they use JUnit rules and they’re basically a set of HTTP requests that allow you to remotely control AM Java code from your code. These custom functional tests are not unit tests. So in the moment, if you start mocking instances, you’re doing it wrong and they are not Selenium UI tests, but some spoilers, I will talk about that a little bit later. So where to start? On the AM Maven archetype, we have provided some, actually two simple sample tests that you can use as a starting point, as a building block to write your own tests. So I have provided here this Maven command to instantiate the archetype. This is out of the documentation. Another important note is that these tests are created as build artifacts so that they can be executed outside of the Maven build process. And all these classes, the class names of the tests need to end with IT. So there is also a filter in the POM file in order to have them picked up. So if we jump into a real world scenario, I have created here the archetype and we can have a quick look into some of the tests. We find them in IT tests, source, main, and here we can see we have the create page IT and the get page IT. So if we have a quick look into the create page IT, we can see that we have a couple of JUnit rules that are basically taking the heavy lifting of things like authenticating against the AM instance, session handling, and so on and so forth. Here the interesting part is this page rule. This page rule basically is executed before and after every test and it will create a page on the author instance, a randomized page on the author instance. Since this test is intended to verify that the create page actually works and this rule makes already the heavy lifting, we are only left over with verifying that the page actually exists as part of the test. Similar in the get page IT, there is all this heavy lifting done by the testing clients and then the test actually does some get requests to some common URLs that we know from the author that should exist. Moving forward, I told you that we can run this test locally. So with Maven, we can run this test locally either against a local running developer instance or also against a cloud instance. An important point here is if we are running this test against a cloud instance, on the cloud instance we need to create a local admin-like user in order to have the test actually being able to log in and perform the actions. So for this case, I have here a small script. If we have a look at it, basically we enter the test directory and we run Maven. So let’s execute this and we will see the output. So basically we are currently compiling the tests and after a couple of seconds, we will see that the test is running. This is actually running against a cloud instance that I have provisioned and provided with a weekend dot site sample content. So we can see here that the create page IT is currently being executed. Here we get the gate pet, gate get page IT and after a couple of seconds, we see that this is successfully executed. I already executed this also as a cloud pipeline. So if you have been working on AM as a cloud service, you should be familiar with this view. This is a production deployment pipeline and here we see the custom functional test. So we could download the log. I already did that. So we can have a quick look into the log that I have downloaded. We see that it is pretty much the same as the console output as we get when we run it locally with added some debugging. So we have here the get page IT and if we scroll down a little bit, we have the create page IT. All in the end also successfully executed. As for the testing clients, I have been mentioning, the testing clients is a functional testing framework for AM. It is built around AM and AM functionality. It is developed and maintained by Adobe, but it is open source so you can use it and it takes care of the heavy lifting of items like handling, authentication, session handling and so on. It is composed of mostly some clients that take the whole interaction with AM and they also provide you some building blocks on which you can extend in order to create your custom clients or components. So for example, we have the CQ client that takes care of mostly interacting with pages, creating trading pages, deleting pages and so on. Another example is replication client. It performs actions like activation and deactivation or the abstract component, which is an abstract Java class that allows you to handle site components. In the demo that I made, I will talk a little bit later about this. So for the purpose of this presentation, I went ahead and created a custom test that verifies some functionality of the weekend.site sample site. So what this component does, let’s jump a little bit into. So we have the weekend.site page and here on the bottom, we have this small author and this component is actually rendered by some Java code that is part of this weekend code. So if we go to the individual HTML rendition, we get, it’s basically the, it contains a name, it contains some occupations and a picture. And there here we have some HTML around this. If we look into the JSON representation of this node, we see the same value. So we have the property name, we have the occupations and we have the file reference. The peculiarity about this implementation is that the Java model of this component requires all three fields. So these are mandatory fields. If any of those are not present, instead of getting the component rendered, you will get this empty diff rendered. So in order to verify the correct functionality as a quick test, I wrote a test that will create a sample page. It will upload the image because we need this image. It will create this component below this page and then it will make a HTTP request to actually get the HTML representation of this component. And assert the correct values of these tests. So if we jump in back to the code, we can go into my site, it test node two, I created a second tree to have this separated. And we have the get by line ID. So very simple. We have again, all these page rules that take care of the heavy lifting. Here interesting is the before rule. This one will be executed before every test execution. In this step, we are uploading the image to the page that has been created already. Later on with these rules, so I have extended the abstract component with a specific component for the weekend site and we use the create method to actually create this component in the page. Then we set some properties of this component, exactly the ones that we’re expecting and we save. So the safe action actually is similar to going into the editor and clicking on save. Then we get a HTML representation, HTML rendering of this component with this line and we use Jsoup to parse it. Once we get the Jsoup parse, we can get the values of this CSS selector. So we have the name and the occupations and basically we assert that the provided values are like the expected ones. If there would be any error in the backend code, this would be rendered differently and the test would fail. So in order to verify that, we can run it. So we are now running the second set of tests. We will give it a couple of seconds. So this is taking a little bit longer and now we see that the Kate page byline IT is currently being executed. Here in this step, we see that it’s uploading. That was too fast. So we had an upload step and now we see that the test actually got executed correctly. To verify that this is actually true and a real instance and not some pre-made things, we can do something like change the expected value and if we now run the test again, after a couple of seconds, this test should fail. Let’s give it a couple of seconds. So now the test is being executed and we see that the test has actually failed. It is expecting test, but it was provided undress. This is the way that you can leverage to create your own validations and this test would obviously also be executed as part of the Cloud Manager pipeline. So let’s talk a little bit about best practices when writing tests. So this test should not rely on existing content. So if you have some content like this image, imagine you do a migration tomorrow and then the test suddenly starts to fail. You might not even have the sample content anymore or you might don’t know why and then you start troubleshooting. So in all overall, this test should not depend on external content. If they require content, like in this case, this image, the test should contain the content and create the content on the fly. Also this test should be important. That means that any modification that the test does on the instance, any state modification should be reverted by the end of the test. This test should not contain external dependencies. So whenever you create a test, it should be self-contained in the jar with dependencies. If that is not the case, the cloud service is not able to execute it. And remember that all tests that verify functionality on a published instance is behind a dispatcher. So make sure that you have the proper rules to access the pages that you’re trying to test. This test, since they are HTTP requests, they don’t have access to OHA console or CX, 0xde and obviously they don’t have access to the Sling API and so on. So as I promised, let’s talk about Selenium UI tests. This is a super nice new feature that we are going to make public anytime soon. Some of you might have already seen hints of UI tests in pieces of documentation and archetypes that have been published. These Selenium UI tests will be available in the cloud pipeline. So you will have a second step about UI tests. They should not replace functional tests. They should extend functional tests. So Selenium UI tests should mostly focus on the UI, on the visual experience of the customer. And the functional tests are more intended to run on the actual backend implementation of your code. Please notice that Selenium UI tests are slower than functional tests, but we already have some hints and some initial work for them on the archetype. So in the same way as for the functional tests, the Selenium UI tests are… You can run them locally, again against a local instance or against a cloud instance. Same scenario, you will require a local admin-like user in order to be able to run them locally, to run them, sorry, against the cloud instance. So let’s jump over again to the console. I have here a small script about the UI tests. Nothing fancy there, just Maven. Very interesting here is I have explicitly set headless mode to false. So in this way, we can see what the browser is actually doing. But if we run this, let’s do this. It will open, it will start a Selenium grid and it will control the browser, in this case Chrome, to perform these tests. For the simplicity of this presentation, I have removed many of the tests and only left one that will upload an image to assets and wait until the processing flag is no longer there. So now we see that we’re seeing the AM instance. It is starting and it will log in with my admin-like user. It will navigate to this assets side and upload an image. So let’s give it a couple of seconds that you’re able to see the whole interaction. Oh, and these are the challenges of the life. We see that it failed. That is unexpected. We can run it again. While this is running, I can show you how it is actually supposed to look. If we go into… So once the test is successfully executed, hopefully this time it works correctly, we will get a report. The test should actually pass correctly. The report looks something like this and it shows you the tests that have passed, that have failed. And if you run it locally, you also get a screenshot of the test and what it was doing when it failed. As for this test, let’s give it a second try. It seems that there is a name conflict on the actual test. So I guess let’s test this. No, and this time it passed. Okay. Things of the life. As I mentioned, the feature is still… It’s going to be available soon, so there’s still edges to correct there. Here we have a couple of references on where you can start and more information on how to start writing your own custom tests, both for the functional tests and also the UI tests. They are already present in archetype and some documentation. I think I have recently seen some more updates on the documentation. I think we can now move forward into the Q&A. So I already had a couple of questions that I was made in the past to kickstart this. Some common question is, do the tests run against author or publish instance? The answer is yes, they run against both of them. You can target any of them. So you can have some tests that go only to the author or some tests that go only to target only the publish instance or like, for example, the product test, the replication. The replication test in its internal will create a page, will activate that page to the publish, and then it will verify for a certain period of time in the publish if that page has been created correctly. If after a certain timeout, this page is not present, it will fail. Do the tests run on dev instances? The answer is yes, but not as part of the dev pipeline in cloud service. So that means that if you’re deploying code to a dev instance or to a dev environment, you should run it locally because it’s not part of the pipeline. And with that, let’s see what questions do we have in the chat. Yes. Okay. So I see the first question was about running the tests in parallel. Thank you, Jörg, for jumping in. So I think that was already answered. Are they planning plans to show, to get a screenshot with Selenium RANDS in cloud management pipeline? As for now, I’m not completely aware of the roadmap. I think it is on the roadmap, but I don’t have the details about the screenshots functionality. For now, I guess you would need to run them locally. So there’s a comment about mocking AM APIs. In the end, the custom functional tests, I would say, is your tool that you have to make sure that your application doesn’t break on a production deployment. If you feel that it’s justified to mock in some APIs, I guess it is perfectly fine. I don’t think it’s the real intention behind it, but in the end, every rule has its exception. So it’s up to everybody to decide this. In the upgrades on our deployments on Proud, if we choose to run functional tests before, will it create test content on production? If I understand correctly, running the custom tests, if they create content and the content is not deleted, the content will be there. The tests are currently running against the stage environment. So it will be part of the mutable content of the stage environment. That’s, again, very important for the best practices to make sure that at the end of the test, the content that is created is also removed. I see some conversation going around unit tests. Yes, so functional tests should not be confused with unit tests. Unit tests is a different step in the cloud manager, in the cloud pipeline that runs before. So the custom functional tests are really from outside HTTP request against, say, already running instance. So Jorg is asking if functional tests are executed against Proud or only against stage. To my knowledge, they are executed only against the stage environment. Thank you very much for attending. I hope you enjoyed it and I hope it was helpful. And you got now more hints and details on how you can write tests and get confidence on the production deployments. Any other questions and follow-up? There is definitely the forum. And if we don’t have any other question, I would say we can finalize this session. So thank you. Have a nice evening, morning, or depending on the time zone. Bye-bye.