Monitoring App Builder applications with Log Forwarding

Transcript

Hello, everyone. I’m Manik Jindal. I’m a product manager on Adobe developers, and I welcome you all to our session on monitoring project firefly applications with log forwarding. With me today, I have Olga. Olga, do you want to do a quick introduction? Yeah, hi, all. So I’m working in developer experience, and really interested in observability in the recent year or so. Really happy to be here. Yes, for those of you who don’t know, Olga is a supreme engineer who is helping the Adobe developers team out with our observability charter, and she’s been a critical part of the team, and we both are very excited to talk about our upcoming feature, which is log forwarding in app builder applications. So today on the agenda, what we have is we’ll start off with a very, very quick introduction of Adobe developers app builder. We’ll then straight away jump into monitoring applications with log forwarding. We’ll talk a lot about logs today, and I’ll then hand it over to Olga for a demo of log forwarding in action, and then finally, we’ll leave behind a few resources for you to get to both Adobe developers app builder, and then also sign up for the log forwarding beta program. So here we go. For those of you who missed the main stage session, and then missed the Firefly overview session or the app builder overview session yesterday, here’s a 30-second intro into app builder. App builder, which was previously known as project Firefly, and that’s why you would probably see me fumbling around. So app builder empowers developers with a complete framework to extend and integrate with Adobe experience manager. Now I really want to focus on the words complete framework because that’s what app builder does. It focuses on all aspects of application development, starting from the first line of code that you would write to debugging your application, to testing it, deploying it, and then finally running it in production. Now our session today here, it’s going to talk about monitoring the application. So it’s very interested in what happens to your application after you are running it in production. So how do you monitor your application? It’s one thing that your application is up, but how do you make sure that it is up? How do you monitor it is something we will try and answer with log forwarding today? So I’d be remiss if I didn’t have an architecture slide on here because I really want to show you what exactly are we talking about here. So here’s a typical app builder application architecture, the kind of application that you would build to integrate and extend a Ruby experience manager and other Ruby solutions. So on the top here, you have a single page application, which is a browser-based application that is using JavaScript to render the UI. Then this application is talking in the back end to APIs, most importantly the APIs that you may build as an app builder microservice that runs on a serverless platform called Adobe I runtime. For those of you who do not know, Adobe I runtime is based on the open source project called Apache OpenVisk. It’s our deployment of Apache OpenVisk. It’s a serverless platform that allows our customers to run their own code on Adobe’s infrastructure. And that’s going to be our main focus for the conversation today about log forwarding. Now within this microservice, you can interact with multiple APIs in the back end. You can call any number of third-party APIs. You can even call Adobe’s public APIs, get access to data storage, file storage, and publish your custom events, all three of which are additional app builder services that we provide out of the box to make your development life easier. But then while you’re calling all of these APIs and accessing your storage, all the magic in the back end is happening in this microservice. And your application, which is basically functions running on a serverless platform, is generating all of this code. And so it’s generating all of these logs. And today we’re here to talk about those logs and how you can get access to those logs in various methods. So let’s talk about monitoring with log forwarding. So before we go in, as I was saying, let’s see how you can fetch those logs that your microservice in the back end is generating, the microservice which is running as a function in our serverless platform called Adobe I Run Time. So there’s mostly, if you take it from a 40,000 feet view, there’s mostly a singular way of accessing your logs today. Now it comes in various flavors. You can play around with the parameters and change the behavior slightly ever so. But ultimately what you’re doing is you are taking a CLI and you’re executing a command that is fetching your logs for you and showing it on the terminal. So the command is called aio-app-logs or it’s alias aio-runtime-logs. So when you execute aio-app-logs, what it does for you is it fetches a limited number of activations, logs from the limited number of activations that your application has. So a typical app builder application would have one or more actions in the backend. And these actions are independently running based on the request they are receiving and then generating these logs. So if you run aio-app-logs, you would be seeing logs from any one of those actions, mostly the recent ones that are just happening, the requests are coming in. So you can see those logs on your terminal screen, and that’s what aio-app-logs does. Now I mentioned something while I was explaining what aio-app-logs does, I mentioned something which is called the activation. So let’s take a step back and understand what is an activation. These actions, when I mentioned a serverless platform, they have an instance of this action being run, one single invocation of this action, of this code, that is called an activation. And it is referred to in our system and even as a developer, you would refer it to as an activation ID. So what you can do is you can use our command line tooling and just say, hey, give me logs for this single activation ID. I know this action ran, and this is the ID for its activation. So you can provide that and see the logs coming out for that single activation ID. Then as aio-app-logs go, you can fetch logs for all the actions in your backend, but then you can also narrow down that search or narrow down that output to a singular action by providing it with the hyphen A flag and saying, just give me logs for this action, the recent logs for this action. And finally, the last thing that you can do here is you can say, give me these logs, but keep pulling them. So you want to tail these application logs. And what you can do is just run aio-app-logs hyphen-tail. So this is, I would say, while this is how things are, it’s a long way of saying that essentially the only way you can access logs today is through the CLI with these commands. And while we know this works fine for application development experience, when as a developer you’re working and you’re building this application, you can use the command line. You would also like to do that. But then for production applications, it’s not easy. You’re not thinking about one single request, how can I see logs? You’re basically thinking about, hey, I have an issue. How can I search for logs that are relevant to this issue? I want to issue a text-based query and say, guys, get me the logs that are relevant for this query so that I can triage and understand what’s going wrong in my system. And that is not something that you can do with the system we have today, with the CLI approach that we have today. So enter log forwarding. Now, what is log forwarding about? So log forwarding is not a new concept. It is a very well-known term in the industry. So what we are essentially talking about here is all the application logs that are being generated by your microservice, we will provide you a way to forward those logs into a third-party log management solution that you own and run. So you have your observability infrastructure which is collecting logs, and you can come in and configure Adobe IOT runtime to send your application logs to the systems that you use, and then see those and view those logs in that system. So that is what log forwarding is about, and that is how it would work. And Olga will show you more in the demo how it actually works. It’s pretty fabulous. But we’ve made this… We’ve talked so much about what’s the current thing and how log forwarding works. But essentially, what is the benefit of you as a developer using this feature? So of course, keep in mind for production, this is something that you definitely would want once your application is live and you want to monitor it. But what are the concrete things that come with log forwarding that you can take advantage of? So there’s essentially three buckets that I like to say, and then let’s go through each one of them one by one. So the first one here is that with log forwarding, you would be able to access all of your logs. And now to contrast it, with the CLI approach, you can today only see logs for activations, the single run of an action. You can only see logs for activations that have either failed. There were some developer error or some application error, or those activations that were asynchronous. So something that you fired and forgot about, and then you want to check the result and see what the logs are. Those are the two cases where you can see the logs. So if you imagine there’s an HTTP request that you’re making to your backend and it succeeds, that is a log that you would not see through the CLI approach because the HTTP request is a synchronous activation. We are waiting for the request response to come back, so we’ll have the result. So it’s synchronous and it’s successful. So it’s a prime candidate. Those logs you would not see through the CLI approach. Furthermore, whatever logs we surface, they have ultimately a retention period of seven days, so you cannot view logs older than seven days. How are we doing on time? Some minutes, I think. Yeah, I cannot see. I’m suddenly like, my phone’s not near me. Okay, perfect. Then the second bucket here is searching through logs. As I mentioned, if you have an application on production, you’re not looking for a single activation or give me all the logs or something like that. You are basically trying to perform a text query, perform an unstructured search into this log data and have relevant results pop up. So once your logs are forwarded to your log management solution, that is exactly what you can do. You can utilize the full power of that log management solution. You can do all sorts of querying, set alerts on logs, and we do that all the time in our teams. And then you can also do some charting, some visualization, parse data from these unstructured logs and then just chart them up. So that is all the power of your log management solution, but it’s enabled by forwarding those logs from runtime into that solution. Now, the third one here is something that we have heard from a lot of our customers is, you have to realize that an App Builder application is not an application that is a standalone application. It’s not operating in a vacuum. It is connecting two or more solutions together or extending a solution. So our customers do not want that solution A’s log to go someplace, solution B’s log to go someplace else. And then App Builders logs are just sitting right there. They’re not going anywhere. So what they would rather do is have the logs from all of these three things combined in one single place so that their operations engineers have an easier time when they are in a situation. They can respond better. They can search through all the logs at once from different solutions. Interesting patterns can emerge. It would help them manage and administrate their log management solutions easier. And it would just overall improve their monitoring posture for their production applications. So those are the benefits of log forwarding. It simply starts by forwarding your logs from runtime from App Builder into your log management solution. Now you must be asking, okay, so which solutions can I forward these logs to? And here’s the answer. So of course we support Adobe IOT runtime that would remain the default way of storing these logs as is today. And those logs would be continued to be served by the CLI. But then you can go in and change that configuration to something else like Splunk. We support both Splunk Cloud and Splunk Enterprise. Or you can change that configuration for your logs to go into Azure Log Analytics, which is part of the Azure Monitor offering. So while you can do that, we are going out with these two solutions, Splunk and Azure Log Analytics in the first go. But we are curious to know from you which other solutions you would like us to support in the future. So if you have a thought about this, go ahead in the chat and tell us which log management solutions you would like to use to forward your logs from App Builder applications into your centralized observability infrastructure on your end. Now in terms of timeline, when this feature is available, I’m sure you’ve already heard App Builder’s GA is set on December of this year. So December 2021. And this particular feature would go live in the coming month after that, which is January of 2022. So this would be generally available. There will be documentation, you can just pick up your terminal and start playing with it to configure those logs, not to see them. You can see them in the log management solution. But before that, we do have a beta program that we are forwarding logs for some customers today. Those are some early adopters. And if you want to try it out today, if you already have an application in production and some of our customers do with App Builder, go ahead. Towards the end of this presentation, I’ll leave you with the resources on how to sign up for the beta program. But right now, I must hand over my time to Olga for a quick demo. Thanks, Monique. Here I am back again. So let me share my screen. Okay, you probably see ourselves now. Here is my ID and let me just make it a little bit bigger for you. Okay, so here I have just a simple project with couple runtime actions. One action is main and what it’s doing is just calling another action that we, the name for which we are passing as a parameter. And if that call succeeds, we log an info message and if that call fails, we log an error message. And another action is very simple, it just returns its own activation ID. Now let’s see how we can work with those actions and how we can work with logs from them, which is even more interesting. So first of all, we’ll be working in a couple runtime namespaces. First one is dev namespace and it will represent the developer scenario. So that’s how developers probably work. So let’s check which namespace we’re in. We can use a runtime namespace list for this and it should show us which namespace we’re in. So yeah, it’s dev namespace, which is correct. And now I’ll show you a new AIO command that allows you to check configuration settings for log forwarding for this namespace. The new command is runtime namespace as well log forwarding get. And it shows a configuration to the current namespace and it tells us that it’s a Adobe AIO runtime. It’s a default configuration for log forwarding and it allows us to work with logs in the same way as we did before through AIO command line. Just to remind everyone or for new people to show how it looks like in real life, let’s do a few invocations of the main action and see how we can get those logs out. So to invoke an action, we run a yearntime action invoke. And as I mentioned, we need to specify the child action that will be invoked there and its child here. First we can try to the result option, which allows us to see that it’s working, kind of return the result. And then we can call it a couple more times asynchronously. Okay, so we did a few invocations and now we can try to use AIO runtime logs command to retrieve those logs. Let’s see. Okay, it usually takes some time for the logs to be reindexed. So sometimes you would need to wait to fetch the logs. But let’s see how long it will take when you are a demo. And also, yes, Monique mentioned you can use activation ID to make it more specific. Or you can specify how many activations you can fetch logs for. So yeah, finally, we got our logs reindexed so we can get them. We got the log from a log record from the latest activation. And with some additional commands, we can or options we can fetch more logs like from other activations or from specific one. I don’t want to go deep into details about that. And rather, I’d switch now to our production namespace to show how we can work with logs in a new way. So let’s switch to our production namespace. We use AIO app use for that. I’m using also merge option. So our configuration is automatically merged. It’s remote configuration merged with our local one in the project. Okay, and now we should be in the production namespace. Let’s just check it quickly. Just in case. namespace list as we use it before already. Yep. So it has no suffix. So it’s my production namespace. Now let’s use the same command to see which configuration for log forwarding is here. AIO runtime, namespace log forwarding get as we used it before in the dev namespace. So as we can see, it’s the same configuration at the moment. So let’s do something more interesting as we are in production. And we probably want to use logs more or like work with logs more efficiently. Now we have another command, which is runtime namespace log forwarding set, and it provides you a few options for log destination. First one is Adobe IO runtime. It’s the same one as we have right now default option, not very interesting at the moment. Another one is Azure Log Analytics with its own settings. And the third one is Splunk. And that’s what you’ll use today. It will ask you now about some settings and I’ll use my temporary ones here. So we specify Splunk host, Splunk port, we are copying index name just to not make a typo and token of course. And now let’s check that it was indeed set correctly. Yep. Now when we call log forwarding get, we see that it’s Splunk hack now for this namespace, we have Splunk host, port and index name. And we don’t have a hack token because it’s a secret. So it’s hidden in the get response. Now let’s do the same thing as we did in the dev namespace and invoke our main action. Here in time action invoke and we’ll do it with result first. Yep, it works. Let’s call it a few more times just to create some traffic here. Okay. So now all the logs from the main action and even from the child action should be going to Splunk. While Splunk is reindexing it, let’s do a small experiment. What I want to do, I want us to get to this error scenario. And to do that, what I want to change is to just delete or kind of undeploy our child action so this call will fail. Okay, what we can do for that, I run time action delete child. And let’s check. We can use I run time action list to see that we did have only main action deployed now. Okay, and now let’s call our action again. So with result, what do we get? An error. That’s exactly what we expected. And now let’s do it asynchronously as well a few more times. And then we’ll have to Splunk and see what we get there and how we can work with that. Okay, so now we can do that. So let’s look at us again. And here is my login demo index in Splunk. And let’s see if we got something. So in the last 15 minutes, yeah, we have some messages here, which is nice. So what we can do in Splunk, first of all, let’s try and search for something. Let’s say you want to search for all the calls to specific action. In our case, it’s child. So what you can do is specify an index and then you can just insert search query and it’s child. And let’s see what we find. Yep, we found all the log messages that indicate that we are calling child action. That’s one thing that we can do. Another thing we can do is the same that we are doing with a your CLI search by activation ID if you need that. So for example, let’s see, you can take this activation ID, and like how you can search, you can look at the log structure, log record structure, and you see that there is some metadata, there is log message itself, but there is also action activation ID, and you can sort by those things. So now, in our case, it’s activation ID. So this is the same, you’ll get the same effect as a your CLI runtime logs, but in Splunk. So like it’s the same functionality, just more. Now, as you remember, we made some errors to trigger the error scenario. And now you can see that those messages could not call child action, go to std error, and we can see it from the metadata in the log records. So those are in std error, and the previous ones are in std out. And here’s what you can do with it. In Splunk, there is a feature of time chart, there are actually many features, and it allows you to see number of log records per specific field. In our case, it’s log stream, it shows us whether it’s std out or std error. And here, you can observe your application, all the pink ones show a successful scenario, because the log records go to std out, and all the purple ones go to std error, it means that there was some error. And we can see the timeline when it happened, as well, we can, as we can conveniently go to specific messages and look what was happening there. So it’s like really nice, you can do zoom and things, things like that. It’s just very convenient. So I think that’s all for the demo. So now you know how you can use Splunk with our log forwarding. Let me find where I can stop sharing. Thank you so much, Olga. It really makes my day to see. I have seen it already five times, and I just cannot, Splunk is by the way one of my favourite tools, so let’s go back to sharing the presentation. So folks, now, how do you get access to this really cool feature? So the first step you have to do, and you must have to do, is that you have to be a part of the App Builder trial program. If you’re already part of the Project Firefly developer preview program, that also works equally well. But if you’re not, then go ahead and scan the QR code on your left, and sign up for the App Builder trial. Now that you are signed up for the App Builder trial, if you want to access the log forwarding beta, you already are building something, and you want to start forwarding logs for that in production or even in developer workspaces, whatever suits you best, go ahead and scan the QR code on the right, and there’s a form that will pop up. Just enter the information and we’ll get in touch with you. Apart from that, this is a great slide for screenshotting. So just go ahead and screenshot that, and the various resources are there. We have a support forum on experience league communities. The team, Man Start forum all around the clock, so you’ll get answers really quickly. We have our developer guide and the documentation, the resources, the links right there. So go ahead, and we have a lot of content on YouTube, Medium, and follow our Twitter handle on Adobe Devs. So that’s it for today, and we can take a couple of questions. I know there was one question around whether log forwarding happens asynchronously or synchronously. Ultimately, does that affect my application? The answer is the log forwarding happens behind the scenes. We have separate mechanisms, and it would not affect the performance of the application in any way. Olga, do we have more questions? Yeah, I think we answered our question twice. Great, it’s the same answer, but no, I don’t think you have more at the moment. Okay, so folks, we can hang around for a few minutes. We are before time, but feel free to ask us more questions. All right, everyone, I think you guys don’t have as many questions, but we are really excited. If you want to sign up for the log forwarding beta, do give us, just fill out the form, and we will get in touch with you really quickly. This is releasing in January of the coming year of 2022, and thank you all for attending. I hope you have a pleasant conference ahead, and a great rest of the day. Cheers.