CDN & WAF Configuration in Adobe Experience Manager as a Cloud Service

Transcript

Hello, my name is Marius and together with my colleagues, Quentin and Florian, I will be presenting the new configuration capabilities for the CDN in AEM cloud service. First of all, let me talk a little bit about the agenda. I will give a general introduction into the CDN configuration. Then Quentin will deep dive into a specific use case of traffic protections and at the end Florian will show you how to use it in a small demo. So you probably all know that AEM as a cloud service has managed CDN in front. We will be talking about the options that were introduced to configure this CDN. Let’s recap a little bit the existing topology and the important features of the CDN. The most important thing to remember is that in AEM as a cloud service, you always go through a CDN. That’s because you will need custom domain and a certificate, so you will need TLS termination, but also for scalability. So the browser always connects to the Adobe Managed CDN and then the request goes to dispatcher and to AEM. For edge delivery, we recently added this ability to place an Adobe Managed CDN in front of AEM live. As you see, the edge delivery has a similar pattern, but it’s also widely used with customer CDN that connects directly to AEM live. But I will not cover the edge delivery part right now. I will focus essentially more on the AEM cloud service configurability of the CDN because the one for edge delivery, the configurability is not yet launched. Until now or until recently, the CDN was not configurable in AEM cloud service and there were a lot of use cases requested by customers that needed this kind of ability to interact with the request and the response as early as possible. These use cases, the transformations that you can do at HTTP layer are typically to remove and add headers, rewrite the request path, block traffic, redirect the client or proxy to different origins. As the CDN was not configurable, this was typically done in AEM or in Apache. So in AEM, you could have written AEM servlet filters. There are a lot of customers that implemented proxies or redirects in AEM. The same can be done in Apache directives. You could set headers like we have here, setting some caching headers, or you could even use more proxy or more rewrite to interact with the request. But of course, it’s always better to do it as close to the edge, to the entry point of the request. That’s why we introduced new capabilities to configure the CDN. Some of the advantages are clear. If we think that the outermost infrastructure is better for a lot of use cases, for example, security, you can stop the attacks at the edge and not let them go through your application at all. Or for consistency, if you want to add a header or remove a header, it’s better to remove it, add it as early as possible so that all layers of the application can see the same thing. Also as I mentioned at the beginning of the presentation, we are moving towards a world where AEM or dispatcher might not even be there. If you want to customize something on the request flow for edge delivery services, for example, you’ll need this kind of configurations at the edge. For that, we introduced a way to deploy rules for the CDN. This is a very simple way of declaring the rules in a YAML file and running a configuration pipeline in Cloud Manager. The same typical patterns of deploying code apply. This is a quick pipeline. It takes a few minutes to be applied and it’s declarative. We have listed most of the sections that are currently supported. Request transformations, traffic filters, redirects, origin selectors and response transformations. I will go quickly through each of these to understand at least the request flow. The browser makes a request, reaches the CDN and then the request transformations are first executed to add or remove headers. Traffic filters are used to allow or block traffic. You can look at the client credentials. The configuration is run and then decide if the client should be redirected to another resource or eventually go to origin if the resource is not in cache. On the response flow, there are only response transformations that are run and that’s the last chance when you can add or remove headers from the response before they are sent to the client. The general structure of CDN rules looks like this. The rules have a name, a condition and an action. The condition is used to filter or to match the particular requests and you can do this match on request properties like the path, the URL of the request, the method, the query string so you can match only certain requests but you can also write rules that have conditions depending on the client IP and some attributes inferred from the client IP like the client country, the client region and also there are some other properties like the tier and the domain. The tier is author, publish, preview so you can target your condition only for one of these tiers. It’s recommended to always guard your condition with tier or domain so that the conditions are not applying to the entire environment. Of course, you could also apply to the entire environment but you have to be certain that that’s the appropriate thing. You can also read request and response headers, cookies, post parameters and also variables are more advanced concept used to pass data between the stages of processing so for example between request transformations and response transformations. Now I will go through each of these stages of processing and describe the typical use cases for request transformations we have as I said earlier they are executed the first ones in the request flow and they are typically used to normalize the URL. One of the most used cases is to remove the unnecessary query parameters for example. So this will maximize the cache ratio because the cache key contains also the query parameters so if you have equivalent URLs that differ only in the query parameters you should remove the query parameters and normalize the URL. You can also strip the HTML for extensionless sites or sanitize request headers. For example, removing authorization header is also recommended on a published site that doesn’t require authentication. Again this is request transformations allow you to normalize your URL space, increase the cache hit ratio and avoid cache busting attacks. Here is an example of request transformation that matches a domain and a path and removes all query parameters that do not match search ID. So essentially it only allows the search ID query parameter that you know is used in your application. Also we have an example of unsetting the authorization header. So this is how CDN rule would look like condition, action, condition, action so multiple rules can be defined. The next in flow will be the traffic filters where you decide whether to allow or block traffic but I will skip this one because my colleague Quentin will talk about this later. And in the request flow the next thing that executes is the authentication. We support only a small set of authentication mechanisms so header based authentication which is used to integrate your own CDN with using an edge key so you define your own edge key, you deploy it and then you can set it up in your CDN to authenticate against Adobe Manage CDN. Also you can set up your own purge key to do purge requests and there is support for basic authentication that is mostly recommended for pre-live environments or lower environments but should not be used as a full-fledged authentication mechanism. After the traffic is allowed or blocked, credentials are checked, you can decide whether this is a resource request that has to be redirected and that can be done with redirect rules. You can see here action of type redirect and you specify a location so this one is doing a redirect from example to www.example.com. The redirects are meant for systemic redirects. You cannot put a very large list of redirects because the entire CDN configuration is only supporting 100 kilobytes so you cannot put 100,000 redirects in this file so for those use cases you still have to use the dispatcher or in this case you need to use the Excel spreadsheet redirect style. After all these steps are executed the next one would be to send the request to the origin. If you don’t define any external origins the request will go to the default origin of the map for that domain so to publish or to author or to preview but you can set up your own request to an external origin. You can rewrite the path with request transformations as I showed before. You can inject secrets so essentially you can connect to any external origin that you have. What is important to keep in mind here is that you can also use the request to the external origin. You can disable it by configuration and there are different cache keys that are created for each origin. Here is an example of how to proxy to edge delivery site. You define the origin and then you can use the request to the external origin. Here is an example of how to proxy to edge delivery site. You define the origin and then writing a condition. If the path matches some specific patterns you select that origin. The last section that is executed is response transformations. You get the chance to set headers that will be sent to the client. What’s important to know here is that only the header and the status can be modified and the body cannot be modified. With that I conclude the introduction for the CDN rules and I will hand over to my colleague Quentin to talk about traffic protection. Thanks Marius. For the rest of the sessions we will be talking more about the security aspect introduced by the new CDN capabilities. We will be focusing on one specific kind of CDN rules called traffic filter rules. These kind of rules let you filter traffic directly at the CDN so you will be able to block, log and allow traffic based on the request patterns. As you can see in that example traffic filter rules follow exactly the same syntax as the other rules that Marius introduced you to. Here you can see how you can leverage traffic filter rules to block specific IPs or a list of countries and how you could even log any logging attempt on your website. With traffic filter rules you can even try to filter more advanced patterns. You could even try to implement your own simple WAF that you would deploy directly at the CDN. Here we have a few examples of how to start. For example you could look for any SQL keyword, the most common one, developing the query string using a regex. That would block any attempt of an SQL injection. You could also analyze request paths and look for traversal path attacks. You could finally also block a PHP extension file because first of all AEM is not using PHP but also because PHP is one of the main targets of vulnerability scanners. It just makes sense to block that. That approach looks nice. It has some limitations that you should be aware of and some caveats. For example following a regex based approach is quite time consuming and it might not be that robust. You will have to look closely and monitor your CDN traffic in order to see if attackers find a way to bypass your rules. You will have to iterate a lot on trying to tune and improve your regexes. There are also some limitations not directly introduced by the traffic filter rules itself but by the underlying language that your rules will get compiled to which is VCL. For instance one of the limitations is that you cannot iterate through request header. If you want to look for an XSS exploit in the request header, you will have to list every request header you want to inspect. A second limitation is around request body which is a big deal because it’s one of the vectors of attack. VCL doesn’t let you analyze and inspect the body of a request. For customers that will need more advanced protections directly at the CDN, for that Adobe actually released a new product at the beginning of the year called AEM WAF. It will run directly at the CDN and offer much better protection against a WAF stop 10, so a screen injection, XSS, etc. As well as protection against anomalies by being able to identify requests coming from scanners or requests towards non-private files. If an attacker is trying to request an environment file or password files, the WAF will be able to identify that. It will also offer better protection against non-bad actors, being able to flag malicious IPs, malicious bots, etc. AEM WAF comes as a self-serve product and with its own SKU, so it means you will have to license it in order to be able to use it. It comes with a pre-configured set of rules that you can activate directly from the traffic filter rules. It’s quite simple. Here we have an example of how you would block a WAF’s top 10 attacks on your website simply by defining a new rule in blocking mode. And then in the WAF flag, you will define any protections that you want to enable, so protection against a screen injection, XSS, traversal, etc. And so here you can see there is no regexes anymore. The AEM WAF will actually own that part. It will be able to identify a malicious request. So it means you will save a lot of time by using it and it’s also far more robust because you can really analyze the whole content of a request. So now that you know about which tool you can leverage to protect your website, you might wonder how do you know from what vulnerability or what pattern you will need to protect your website from. And so for that, Adobe also provides tools in order for you to better analyze and understand your CDN traffic. So those tools are available in two variants. The first one is an Elastic Circ key banner stack that you can run locally on your laptop. You will download your CDN logs directly from Cloud Manager, import them, and then you will have access to the following dashboard. And the second variant is for customers who forward their CDN logs toward their Splunk infrastructure. And so for them, we provide also a Splunk dashboard that they can install in their own Splunk instance and they will have access to the same level of information. So I won’t cover all the aspects of those dashboards because we actually have a lot of tutorials on the Adobe Experience League, where you will be able to learn how to ingest your CDN logs, how to use those dashboards and come up with the right traffic filter rules to block malicious patterns. But in a nutshell, you will find the most important metrics such as top IPs, top URLs, top user agents. You will also find distribution of traffic over time. And of course, if you are WAF customers, you will be able to get access to WAF insights directly from those dashboards. So you will be able to tell what kind of vulnerabilities attackers are trying to exploit on your website. So now let’s try to focus on a specific kind of attacks called DDoS attacks. So nowadays, any website can be the target of a DDoS attack, unfortunately. And so for that reason, Adobe provides some out of the box protection against DDoS attacks. Some of the protections are directly offered by our CDN providers that will protect your website at TCP IP level. And on top of that, Adobe provides HTTP level protection, mainly leveraging rate limiter. So it means any AEM as a cloud service environment is equipped with a rate limiter that is configured the same for all environments and that is meant to block volumetric attacks. In the unfortunate event where our protections were kicked in, we would directly notify you via Action Center. So you will be aware that your environment, your website is getting attacked and that the out of the box protections kicked in and tried to mitigate the attack. In the case that it happened, we highly recommend you to download your CDN logs and use the dashboard I just presented to you in order to better understand the attack and try to come up with traffic filter rules in order to mitigate the current attack or future attacks. And so what you can do in order to protect your environment against DDoS attacks is to actually leverage the rate limiter feature from the traffic filter rules. And it’s also. So here we have an example how you would limit the number of requests per IP. As you can see, it’s super simple. You will define a normal traffic filter rules where you will have a rate limit property that contains some sub property. The first one limit that goes with window, which is basically the average number of requests for which the traffic filter rules, the rate limiter would kick in. Then we have the penalty property, which is basically for how long the IP will get blocked if it triggers a rate limiter. And then we have the group by property, which is set to client IP. So the group by property can be used to basically aggregate the counter. So you could count requests per IP obviously, but per user agent, per path, per country, etc. And finally, we have the count property, which can take one of the following values. So whole, which means we will count every request that goes to the CDN, fetches where we will count only requests that goes to the origin. And finally, errors that would count only client side and server side errors. So we highly recommend you to use the free variants because they will protect your environment from different things. So the whole will protect mainly your budget, your request budget. The fetches will protect your origin. So it’s the most powerful one. It will basically prevent an outage. And finally, the errors one will protect you from different things. So, for example, if you have a scanner that tries multiple paths, they will likely end up as a 404 that will count against the error scanner. And they can also be used to block IPs that are making a lot of requests that end up as 500, which could be the early sign of an outage. So we highly recommend to use three of them. And to start with a high limit for the count all, a lower limit for fetches, and finally a quite low limit for errors. Another important notion here is the alert field and the action. So basically, if alert is set to true, you will get an action center notification every time the traffic filter rules is getting triggered. So a question we often get asked is, what should be the limits? And so to be honest, it’s quite hard to tell, because every website is different and it depends also on your requirements and needs. But what we highly recommend is to actually use a dashboard where we expose the maximum number of requests per client IP for all, for fetches and for errors. And here you will be able to tell basically what should be the limit. So it gives you an accurate limit based on your own data. Now, we actually wanted to give you some tips how to best protect your environment and your website from DDoS. And so the first tip here is to ensure fast response from origin. So try to not have like responses that takes up to one minute to generate because it can be seen as a vulnerability from the point of view of an attacker that tries to DDoS your website, because you will have to just send far less requests towards that endpoint and it will likely lead to an outage. The second tip is to leverage good caching. Because basically, if your website is well cached, it means a DDoS attack will be able to be absorbed by the CDN. It means requests won’t go to the region, so it will reduce the likelihood of an outage. And a good thing to do here is to actually leverage the request transformations in order to remove unknown query parameters. So it will improve the cache key and it will increase the likelihood of a cache hit. The next tip is to use Dashboard. Dashboard will show you the most important metrics, but it will also give you, as we saw earlier, an accurate value for your rate limiter. The next tip is to use different strategies. So it’s quite important here. We saw it with the rate limiter. You could have three different rate limiters that protect your environment from three different things. But another good strategy we have seen is if your website is only doing local business, like, I don’t know, for example, only for Switzerland, then you might want to introduce several rate limiters, one for users based in Switzerland and another more restricted for users based outside of Switzerland. Next, don’t forget to set up an alert because it’s actually the only way for you to get notified when your environment is under attack. And final tips, quite important here, with traffic filter rules, always start with action log. And then once you are sure that the rules are behaving as you would expect, then move it to block. Traffic filters are quite powerful. And yeah, if you are not cautious enough, you might actually block the website totally. So with that, I now hand over to Florian for a quick demo of the CDN configurability. Thank you for the handover, Quentin. So let’s go to the demo part. I have a pre-recorded demo in order to also show you the speed, but also to be able to speed up the recording while we see it. So like this is the normal weekend website that you know from many of our other examples from Adobe Experience League. What we have here is, what we want to do here is to show you an example of the traffic filter rules as well as redirects. Traffic filter rules are meant, as Quentin already mentioned, to block your site on multiple different request properties. For the sake of this demo, because it’s easier to demonstrate, we will use the path so that actually just certain parts of the website are blocked. This, in a real life scenario, would be, for example, blocking by country or other request properties. For the redirects, we have a redirect from slash about to the about page of the US specific part of the website. In the real case scenario, so Marius mentioned this, you would use, for example, redirects to redirect from your Apex domain to the www domain. Or, for example, also to redirect from your Apex domain to the country specific page, for example, then to US English, etc. that you can do at the edge as well now. In order to create the pipeline, we go to Cloud Manager. It’s a new pipeline that is a non-production pipeline here in our case because we have a sandbox. It’s a deployment pipeline, so similar to the Web tier pipeline and the front-end pipeline that you already know. You have to give it a name. You can start it manually or automate it from Git. You have to select your environment, then you select the Git branch that you want to deploy from. And then the configurations for the CDN are in a special folder, normally slash config. That’s what we do in the demos, at least, always. And this is, for example, also what was in the example that Marius showed. Now, when we run this pipeline, this is usually taking roughly one to two minutes. So for the sake of this demo, later on we speed up the execution a little bit. And you can then later on, for example, also download the logs in case you have some errors. So in case you see some errors in the UI here, you can always download the log file and this will show you where possible syntax mistakes, etc. are. For Cloud Manager, you have to check in your code into your Git and then you can execute it from here. Now we can see the pipeline is already done, but it might be that the configuration is not yet deployed. And this is because actually the code needs to be deployed to multiple different POPS in Fastly. And this might actually take a little bit of time depending on the POPS. But usually this is available within, yeah, also like one or two minutes. It might take up a bit longer, but yeah, usually it’s one or two minutes. And as we see now here, so this is the use case that a potential attacker would see. So you get a 4.6 error and we can also verify that our redirect to the slash about is working and is leading us to the yours about page. If we don’t want to use Cloud Manager all the time, we also have the option now to actually deploy CDN configurations from the RDE. So this is a normal RDE environment that we have here. I pre-installed already the Wicket bundles and the Wicket dispatcher configuration. So everything is set up in that way. And this demo is actually to show you how to quickly develop without checking things into Git. So what we want to simulate here is that we have a typo. For example, we have the redirects specified in a wrong way. So the property is not as it’s supposed to be. And we can install this with a new type for the configuration for the RDE, which is the end config. And as you see, this will take roughly 30 seconds or so to come back. And we can see here the error is directly displayed. This is something that you would see otherwise in the Cloud Manager logs. We can see redirects to is wrong and redirects should be the right way to go. We can go back now again and redeploy the code again. So for this part of the demo, I’m not speeding up anything here to actually show you the real development speed, how you could actually develop on your local machine by yourself and on your own environment. It varies sometimes a bit on cluster load, etc. as well. But usually CDN configurations are deployed within one to two minutes to the CDN. And then at the pop, it might take another minute, as you’ve seen before. So this is, of course, the demo effect that it takes a little bit more longer. We should soon be seeing the four or six as well, which means that the adventures page is blocked as we defined in our configuration. OK, here we see this and then the slash about redirect is also working as expected here. That’s it so far from our demo as well. Thank you very much for joining our talk. And yeah, I hope you enjoyed it.