Using Sensei to take Analysis Workspace to the next level

Transcript

Please welcome to your screens, Mathias Kolić. Hi everyone, welcome to your next skill builder session. In this 45 minute session we will talk now about how we can use a Dobi Sensei, our artificial intelligence, to take analysis workspace to the next level. My name is Mathias, I’m a technical trainer at Adobe. I train Adobe Analytics, Target, Audience Manager, also the newer Adobe Experience Platform and a few other tools. But this session is all about Adobe Sensei. I have a short little agenda for you. So at the beginning we will jump into the anomaly detection in the workspace. We will go to the next level after the anomaly detection and talk about the contribution analysis. We will create an intelligent alert together. We will see what does the intelligent alert has to do with the anomaly detection again. We will talk a bit about the segment IQ, what we can do with segment comparison, how we can use different segments in fallout visualizations and so on. And at the end of the session we will also have a brief look in how we can use an algorithmic attribution in our attribution models. But I more want to tell you when do we use the algorithmic attribution or when do we have other attribution models we can use in analysis workspace. Let’s jump into the content. We will start directly with a short overview about what is actually Sensei. What do we mean with the Adobe Sensei? I’m pretty sure you heard that in different contexts with different solutions in the Adobe world. At the end of the day, what we face at Adobe is that designing and delivering the perfect customer experience can be complicated. And especially in the future, it will also become more and more complicated. And with Adobe, we also use an artificial intelligence and machine learning to make things easier and also be able to go beyond just normal analysis or beyond just normal personalization and do the next step for our customers. A Adobe Sensei is not just Adobe Analytics. You might have seen already other solutions as well. I do later on an Adobe Target session. We also talk again a bit about Adobe Sensei because we can also use it for personalization. We can use it in the audience manager. I’m pretty sure it’s also even used in the Adobe Experience. You had some sessions about today. And even in the Creative Cloud, we have Adobe Sensei available. Let’s have a look at what they can do in Adobe Analytics. And first of all, we have the anomaly detection. So if you look at the data, you will analyze a lot of trend visualization. You might discover that something looks a bit odd or suddenly the data is going down where you don’t expect it to go down. So in the past, we always had a lot of trend visualization. That also means by I or normal human being, you might not discover every single anomaly you will have in your data. And that is where the anomaly detection is coming into the game. Transfer questions like, what is the anomaly detection? What is the anomaly detection? What is the anomaly detection? What is the anomaly detection? What is the anomaly detection? What is the anomaly detection? What is the anomaly detection? What is the anomaly detection? What is the anomaly detection? What is the anomaly detection? What is the anomaly detection? What is the anomaly detection? What is the anomaly detection directly in the interface? So you should see my browser. Perfect. You can see here already my analyze workspace project. At the end of the day, we have a blank reform table. We will of course not go through all the single features we have in analysis workspace. To just show you a few things about the anomaly detection, I want to keep it quite simple. And I will just throw a revenue metric in this table to come up with a visualization over the time. If I now have a line visualization, to have a look how the trend is behaving, with only a few days of data, you will see directly I have here my line visualization with some strange dot over here. And I also have here in my table a specific marker. I will just actually increase the time range so that we have a bit of more data to look at.

You can see at the moment, it was a bit fast, but at the moment when I change my data range, and I change something in analysis workspace, you see that automatically for all visualizations, the anomaly detection will run. So there’s nothing you have to do to get an anomaly detection that will start automatically. And I can see here with a mouse over a small little hint that a new anomaly is found. In my table, these anomalies will be flagged with that small little black triangle on the top right corner of a specific day. So I have an anomaly here on the 29th, on the 30th of April, where the revenue doesn’t look as expected. In the visualization, you will see here the dots over there. That’s our anomaly detection. So some anomalies you might discover with your eye. So perhaps if you have a trend line or something like that, you will see that these two dots are over the trend line. Other anomalies like this one here or that one might be much harder to detect because they are very near the expected line. So with the poor eyesight, it’s hard to discover that anomaly. What is happening in the background? Why do we have the anomaly? So what we are doing with multiple different methodologies, different algorithms, we run through the past of the data and check what should be the expected value. You will see here in the background where we laid a separated line. That is the expected value for each day. So you can see here at the back, a separated line. That would be the expected value. The anomaly detection, the algorithm will expect us to have that specific day. The actual line, that’s the actual value. And then we have an area in between here in the background. This background here is you can imagine that is our standard deviation. So that means as long as the value is still in this area, to be saying it’s OK. It’s not exactly the expected value, but it’s still good enough for us. At the moment, the value will not be in that area anymore. You will get an anomaly shown over here in Analyzer’s workspace. I will share a few links afterwards with you, but you will find also in our help documentation, everything in more detail about, for example, the anomaly detection. And you can see here is not just one algorithm which is running. We have here five different metologies which are running through the back end of data and detecting anomalies. How far we go back in the data is depending on how big the time range is. So that means the longer the time ranges we are looking at, that also means the longer the anomaly detection has to go into the path to find or to check or to calculate it. Right. Expected value and make sure that we show an anomaly with the high confidence. So the anomaly detection, if you go back to the slides, helps us to see all there is a specific anomaly. We found an anomaly. If you go back to the slides. If you have the anomaly detected, the next question would be, why do we have that anomaly? But in the past, with our web analytics experience, we went through all the different dimensions which could potentially have something to do with that anomaly. So we can analyze different technologies, mobile, desktop. We can analyze specific marketing campaigns. We are responsible for these anomalies in the data. But that’s something we always have to know through our experience or through the analysis we did in the past with your company’s data. What dimensions are relevant? And it might be that it cost us a lot of time to actually have analyzed all the data. And there’s also no guarantee that we will find the right reason behind that. For that reason, we added something which is called the contribution analysis. So the contribution analysis is a feature which comes on top of the anomaly detection and helps us to go deeper behind the scene. Why did that anomaly happen? What was the reason for it? And that also helps us to find them later on the actions we have to do now. Because if we look at data, that’s nice to do. But without having some actionable insights, it might also be a bit of waste of time. So for that reason, we have that contribution analysis. So if you jump back into the interface, because it’s just much more fun to show you that contribution analysis directly in the browser. So if you jump in the browser, I just want to run a short little contribute analysis directly with the data we have here in workspace. And you will see how it works. If you jump back in the browser. We have here the different dots. And if I go to a mouse over, I can also see how big is the anomaly? How much percentage is it above expected? Or in that case over here below expected. So it might also be worth it that I start with the anomaly, which has the biggest impact. That means which is the most, the farthest away from the expected value. So if I click here on that anomaly I detected here, and I click on analyze. I will come to a field which is called one contribution analysis. So if I click here on running the contribution analysis, that takes a few seconds. But afterwards, you will see that there is popping up a few visualizations, a few tables just directly out of the box. While the contribution analysis is loading, so it shouldn’t take too long. That might be very fast because we need a lot of power. Because what we do in the contribution analysis, we will go through all the data you have in the past in your interface. We have some limitations for this contribution analysis. So in the limitations field, every time you run a contribution analysis, you need a so-called token. And you can see here even in the ultimate package from Adobe, we have 20 tokens per month. There are all the packages where we have even unlimited tokens. So that means this contribution analyzes data small feature for the web on a list to really look behind why we have the anomaly. And because that needs a lot of power, we have to limit it to specific tokens. So you’re not meant to run the contribution analyzers for every small little anomaly you have. It’s better to consider the most impactful anomaly and then run the contribution analyzers.

If you run that, you will get total revenue for the time range you have. You will have here the same overview visualization again. And then we come into the addressing part. We will get here top items with a contribution score. That means the nearer we are at one with the contribution score, the most likely this dimension or the specific value in the dimension will have something to do with our anomaly. We have that for dimensions. We also have that for segments. So what the contribution analysis is doing on a next level, it’s even combining different information from our dimensions and create so-called out of the box segments. You can see how a segment is built over there. So that are quite complex segments which are adding different values together. And this combination over here will have most likely to do something with the anomaly. Now a few of you might ask yourself now, why is material showing us that one? That looks all not very helpful. I agree with you. We have here day of year, day of months. So if the anomaly is on the 31st, of course, it’s very, very likely that the 31st have something to do with the anomaly. So what we normally do with the contribution analyzers, and that’s a big recommendation from my side for you, is we are filtering specific dimensions out we don’t need. So it’s a good practice to filter all time-related dimensions out. So how can I do that? I click on edit my contribution analyzers. And before we run the analyzers, we have to field where we can exclude dimensions. We can clear all the dimensions we have excluded from the contribution analyzers. And we can also set a specific setup as our default. So for example, in your real world use case, you will have some data dimensions, some data parting, and might be other technical help for your contribution analyzers.

And for that reason, we can exclude dimensions. To exclude a dimension, we have to search that dimension. So I already have here day and day parting, for example. And if I drop my dimension, which doesn’t really make sense to analyze in the contribution analyzers context, I will very easily and fast be able to exclude, for example, my day dimension, my day parting, so maybe I also want to include day of year, day of week, and so on. So it might be that we have to do a bit of a setup the first day time we are doing it. But remember, you can set that as a default, and that will always be the dimension you exclude from your contribution analyzers. And if I just rerun it with less dimensions, you will then see that hopefully the contribution analyzers makes more sense. There’s also no guarantee if you run the contribution analyzers that you will directly get the right dimension to analyze. But even if in our 45 minutes context of the session, it might take a while to talk about the contribution analysis.

Of the analyzing our data in the real world, it really, really saves a lot of time. So let’s wait a second until that analyzes run again. And you can see that a few of the mixed dimensions are not already filtered out. And then we have maybe more interesting visualizations, like for example, a specific region is very likely connected to the anomaly, then a specific mobile manufacturer might be that is a bug with that specific mobile manufacturer, and so on. So we get all our different dimensions. And if we filter some dimensions out where we know they don’t make that much sense, everything, for example, which is time related, you will get over the time, a more and more reliable contribution analyzers. And again, the new writers at one that means the more confident the algorithm, the more confident Adobe is that this specific mixed dimension has something to do with our anomaly. The lower it is, the less confident we are that it has to do something with our anomaly.

And you will also have the combinations of segments. And here you will just get a ranking of I think the top 26. So my use case of builds up to 26 segments in a real world scenario, there should even come up more segments. With these segments, you will have a specific combination. And especially for this automatically graded segments, it’s even more important to filter dimensions out. Because before we had a segment here where we only had day of months, day of year, day of week, some day parting combinations. And we don’t necessarily want to have that because we don’t do it later on something based on the data. But something like search engine single page visit is enabled. So that means a lot of people who are having a lot of people who have done or who are part of the anomaly also had a single page visit. So that’s also something we have to consider in our analysis. I mean, if they only show a page, it is very unlikely that they can do a revenue unless for some strange coincidence, this page would be the order confirmation page. So if I have a huge amount of single page visits, it might be very likely that I have less revenue because I can’t really make revenue on that site. So I still need to have some context and knowledge about my website, but it can fasten things up a lot for you. If you have to able to run the contribution analysis instead of starting with different device types, go to different browser, different campaigns and so on to figure out by yourself why we have that anomaly in the data.

Let’s jump back into slide deck and go into our next topic. Where we have a Doobie Sensei involved. And these are intelligent alerts. I’m not sure how many of you are building alerts on a regular basis or how important alerts are for your company. My personal point is a really cool feature to have the possibility to build alerts as long as we are not overdoing it. I would say every one of you might have already been in the situation that you get a lot of automated emails in your inbox. And if you create an alert and you send it per email, it’s nothing else like an automated email. So that means that I’ve seen that in the past when I worked as a web analyst at companies that I came to colleagues desk, who had all the alerts or all the dashboards which were sent out by me, this rule packed in a file in a folder in the Outlook and never opened it again. So that means if you send too many alerts and you annoy your colleagues with alerts, it will also end up with nobody else is looking in alerts. But let’s have a look how we can use intelligent alerts to actually get the most out of it.

With intelligent alerts, we have to be aware how the anomaly detection is working. Because if you use intelligent alerts, everything will come back to the anomaly detection. And the anomaly detection, depending on what time we choose in line visualization, that is depending how far the anomaly detection goes back with the analysis. In terms of alerts, it’s depending on what granularity we are choosing. So it means how often do we want to have the alert sent out. And if we want to have it only once per month sent out, the data which is analyzed for that intelligent alert, the last 15 months, last the same range last year, just to get that expected value for our anomaly detection. In the list that here, the other granularity is as well, just in case.

With the intelligent alerts, we work with so-called thresholds. But before we go in detail about that slide, I would jump into the browser. There’s just two things I want to go through here, and that is the example. So just keep in your mind, we talk with intelligent alerts about thresholds. And please keep in your mind for now that if we have a 95% threshold, we talk about two standard deviations. If we have a 99% threshold, we have three standard deviations. And if you only have a 90% threshold, we have one standard deviation. Let’s just keep that in our mind. That sounds a bit abstract, but if we jump now back into the browser, it makes much more sense in a small little bit. So let’s go back into the browser for now.

I click on components, and I create an alert. I don’t want to go through all the features in the alert builder. You always have to put in a title to be able to save an alert. And here on the bottom, we will define when should alert be triggered. And very important, if you track and drop metrics here, every single metric we are dropping here will trigger an alert. So please consider that if you build alerts in the future, any of these metrics are triggering the alert. So it means if I put here a revenue rule, and I put in, let’s say, a message rule, both of these metrics will trigger an alert.

First, it’s okay if we keep it with our revenue so far.

So if I say I want to trigger an alert when the revenue has an anomaly with a 99% threshold. So we said 99% threshold. That is three standard deviations. So that means if you think back about our line visualization we have, we make the area bigger, where we say, okay, if the value is in that area, we will not have an anomaly. So that means the higher the threshold here, the wider is my area. It means the less alerts we will send out. So if I remove that or reduce that, for example, to 90%, you will see we have five times an alert sent over the last 30 days. So that is the important thing about that threshold. If we use the anomaly detection to send an alert also. We say we want to send an alert when we detect an anomaly in the revenue, and then we will just define how sharp we go into that, how wide we build up our standard deviation to say, okay, this value is still okay. This value is still okay. So the higher the threshold, the higher the standard deviation, that means the less alerts we will send out because we will have less anomalies in relation of the anomaly detection. We can also say we only want to have the anomaly triggered if it is above expected, so higher than the expected value, while we only want to have the anomaly triggered if it’s below expected value. So that means it’s lower than the separated line, the expected value we have in our visualization. And of course, that is out of scope for that session. I can also trigger alerts in a more manual way, above or equal, below or equal, or changes by percentage. And then I will give a hard number in when I actually want to have an alert. But if you really use the anomaly detection for your analyzers, the next step might be to also use the anomaly detection to send out alerts. We have here different time granularities, so you can really drill it down to hourly. That means then the anomaly detection will go back the last 336 hours to calculate the expected value. So in that slide, you could see the overview of the different ranges we have to go back, depending on what granularity you are choosing over here. And then with the recipients, I can send an alert to different people who have an Adobe account. You can also send alerts per email. Or if you really want to know your colleagues, you can send the alerts on a mobile number. So I can just put the mobile number in, put it on here on an hourly basis. And that means every single hour an anomaly is happening and someone is getting their alert sent to your text message. Please consider that our anomaly detection is not working in business hours. So the anomaly detection doesn’t care if it’s in the morning at two o’clock. So please be a bit careful with sending alerts to mobile numbers of your colleagues, because you might get some attention very fast. But the receiver also sees who has created the alert. But I just wanted to show you how far you also can go down there with sending alerts out. And you can definitely use the anomaly detection to have an alert if the anomaly detection is detecting an anomaly in the data. Perhaps the last thing from my side, I would recommend you to not do that for every metric, because that will end up with you getting a lot of your own alerts and you will also not really bother anymore. So use that intelligent alerts for your most important metrics for your main business goals, so that you are the first person who knows that there was an anomaly detected in the data. And then you can go back into your workspace project and you can use the contribution analysis to get the first idea why this anomaly actually happened. These three areas, anomaly detection, contribution analysis and intelligent alerts all hanging together, as you could see, because we always use the similar technologies with the different algorithms the anomaly detection is providing us. So let’s go back into the slide deck. Yes, by the way, just one thing to add about the anomaly detection. If we have thresholds of 99.75 and 99.99%, these thresholds are introduced specifically for the only granularity. In our example, you could see even in the only granularity, I only had one alert per day, so it was not that bad. But if you have a huge amount of anomalies detected on the only basis, you might want to change the threshold to even higher than just 99%. So we talked about the first three chapters, anomaly detection, contribution analysis and intelligent alerts, which all have to do this to be sensei and where the algorithms are running in the background. I have two more chapters I want to go through with you. That’s the segment IQ or the anti algorithmic attribution. With the segment IQ, we have two different parts. We have the segment comparison panel, and we can also compare segments in fallout visualization. I mean, segments and fallout visualization, I would say it’s not that much to be sensei because you will just track and drop different segments in a fallout. And then you can compare the different fallouts you will generate. One thing which might be used not that often already on your site is the segment comparison panel. So we have a panel here which is called segment comparison. And it works a bit like the contribution analysis. This big difference that you don’t have tokens for the segment comparison. So you can run as many segment comparisons as you want. And it’s very easy to set up because you only have to add a specific segment. You want to compare against a different segment. So I could say, for example, I want to compare my USA visit was this always it.

That normally shouldn’t take as long as the contribution license. So I would assume it’s done a second perfect. This the comparison, United States versus all visits, but the whole thing behind that is the first question is how much does it make sense to compare this two segments with each other? Because, of course, all the visitors from the USA philosophy part of all my visits in general. But I will get the total number, the different segments, and I will get an overlap number as well. And then it becomes a bit more interesting. We will have the top metrics against segments. We will have here the top dimension items against segments, and even it will use all the segments we already have to compare these. These two segments we have chosen, considered to all the segments we have. So the more different the data is, the higher the segments were ranked. I will come to that in a second. Why it doesn’t make too much sense in my specific trading environment to analyze that further. But the interesting part is more over here. So if I have your two different segments, in our case, the visitors who came from the United States versus all the visitors I have on my website. And I can have here different engagement metrics. So what I can analyze here compared to all the visits I have, how engaged are my USA visitors. So you can see in terms of returned revenue per visitor, in terms of offline revenue per visitors, they have much higher data than all visits. In terms of the whole video part, you can see that the people who came from the USA to our website are not that engaged with videos as all visitors I have, or the average of all my visitors I have on my website. So you can analyze specific segments in more context. And it is even from the use cases we have the segment comparison. It’s very often that we use a very specific segment and compare it with all visits. Why are we doing that? Because we can then analyze in terms of the complete visitors I have on my website. How special are these people from the specific segment I’m comparing. And if the people are very different to all my visitors overall, then I might want to think about perhaps give them a specific experience or do something different on the website for them and use them for example, experience targeting or other options to personalize the website more for these people. And that’s the same here for the top dimensions. I can for example find out, sorry, here, this top dimensions if the people who came from the United States to my website use a very specific marketing campaign or very specific channel to come to my website compared to all visits I have. And here’s also ranking the higher the difference score, the more different the two segments are. So the more special in my case is that specific user group of the people coming from the United States to the website. You have a few visualizations about all the data, but the most interesting part in my personal point of view is definitely a different score. But you can really find out how specific is my segment I want to use. And because I know I’m pretty sure you had already a session or a few of you had already a session about analytics for target. And if we analyze tests for me to be target in Adobe Analytics, we can theoretically also use the segment comparison to compare each of my different experiences versus the control group. So you can really get here very fast, quick and easy overview interface for comparing an experience versus the control group. So you can use it for different reasons. And I will share later a link with the chat where you have some use cases for the segment comparison. But we have analytics for target best practice sheet, where also the segment comparison is actually used to get very fast out of the box, a very good comparison of new experience versus the control experience. So you can see here you just do two clicks, you just create and choose one or two segments, you can even use more than two segments, and you can run a segment comparison, and everything will happen in the background. I will also share link with an overview about segment comparison, but there’s a bit more detail about the technique, which is used on Adobe Sensei to provide that same for the contribution analysis. We also have an audience clustering that’s more or less the part where we have the segments in the contribution analysis automatically created, and also the possibility to compare here the segments you already have in your interface in your segment comparison as well.

And the last topic I have is the algorithmic attribution. If you talk about algorithmic attribution, but what is attribution at all? There is attribution models, we have last touch, we have first touch, we also have like shown here in the screen, you can see a participation model, where you have somewhere on the purchase page $100 revenue, and this $100 will be equally split. So it will not equally split it, every single page with this involved will get the full $100 credit if you have participation on. The algorithmic attribution is just bringing that to the next level. So what the algorithmic attribution is doing, it is assuming that you not have a equal split. So not every touch point should get 100% credit. Or you can also not use a linear attribution, because that would mean if you have $100 revenue, and four pages involved, each page will get $25 credit. So the algorithmic attribution is based on the assumption that we have complete different credits among players with a complete unequal contribution. And to have less fear means we need the algorithmic attribution, the algorithmic attribution in analysis workspace.

I’m sorry, I had already a table for that. Let me just build a short quick table.

So if I want to have a diffusion model, I can use for example, my marketing channels. And revenue. We will see in the last touch approach, how much revenue is coming in from the different marketing channels. And here on my accepting wheel, I can change very fast and easy the diffusion model for my marketing channels. So I can choose here different attribution models, like last touch, first touch, linear participation, the ones I mentioned, and you can also use algorithmic. And if I use algorithmic, you will completely take it out of your hands, and you let Adobe Analytics decide what the anomaly, sorry, not the anomaly, the Adobe Sensei decide what attribution you want to have. There’s a full page about best practice. The most important thing with the algorithmic attribution is normally you might not run the algorithmic attribution if you already have found the perfect attribution model in the, within the normal attribution models you have. Only if you already investigated in different attribution models, and the attribution still doesn’t look right for your business, because you have to completely unequal contribution, then it might be worse to jump into the algorithmic attribution, but you can’t really look behind it. You can’t modify it or customize it. You just turn it on and let it run. So from the best practice side, and I will just jump back to my slide deck for a last time. From the best practice side, you normally have first some analysis. So you want to analyze your data first and see what attribution could make sense. Then you try with your rule base attribution to find the right attribution model. And only if that doesn’t work, then the next step would be to go to algorithmic contribution. So we talked about a few different to be sensei features. We are slightly over time, but we talked about anomaly detection, contribution analysis, intelligent alerts. All three of them come more or less together as a package. And then we have the segment IQ and the algorithmic attribution. It might have been a bit more technical. I hope you enjoyed the session. Thank you for attending. And of course, we have still some time for questions. So now is the moment. If you have any questions, please let us know in the chat. Thanks, Matthias. We currently don’t have any. So if anyone does have any questions, if you just want to pop them in the chat and we’ll be able to answer them.

Thank you.

So that information, but if you have any questions, please let us know.

We still don’t have anything. So I think you answered everybody’s questions. I will share some links in the chat. So might be able to share them then in the chat room in the group. So just want to share with you the links also shared during the session.

So the links, you will find a few use cases, a few best practice for intelligent alerts and also for segment comparison. So I really want to encourage you, investigate by yourself, try the features and you might not have directly the perfect idea after you run your first anomaly detection or your first contribution analysis. But as familiar you will become with the 2B sensor and the features, the more it will help you in the future.

Thank you.

There’s still no questions, so I think we are good to wrap up. So if you don’t have any questions, that’s also for my side. So I hope it gives you a first good overview about what you can do with the 2B sensor features in Workspace. Have a good rest of the day and I hope I will see you in the 2B target session as well.