Rockstar Tips - May 2023 APAC Adobe Analytics Skill Exchange Grow Track

Transcript

Hi, and welcome to Rockstar World Tour Tips and Tricks from the 2023 Vegas Champion. I’m Jennifer Dungan and I’m the Optimization Manager Analytics for Torstar. And some of you may or may not have heard of Torstar. We are one of Canada’s largest publishers. You may have heard of the Toronto Star, which is our flagship product, but we have many other newspapers and digital ventures within that space. I’ve been with the company for about 15 years. I started as a web developer, I moved into QA, and then I finally took over the web analytics. But throughout all that time, I’ve worked with Adobe Analytics at every stage of my career. And in my current position, I also manage the Google Analytics and Parsley Analytics, as well as all of our pixel trackers for our marketing team. In addition to being passionate about analytics, I enjoy reading, playing video games, and DIY projects around the house. I’m also an avid crafter, I do crochet and amigurumi. Here’s a few of my items that you can see. Okay, you can tell I’m a nerd. I get that. Hopefully you recognize some of those characters. And then I also make Halloween costumes, both character based and original. So basically, nerd. I get it. Over the last few years, I’ve had the pleasure of being a three time Adobe Analytics champion, and the co leader of the Gnome East Analytics User Group, and more recently an Experience League Community Advisor. Now, while I get that you guys probably aren’t in the market for looking at North American user groups, there are other user groups within your area. So I highly encourage you to go and check those out, because everyone has such amazing content. And it’s a great place to learn and to pick up new skills. But of course, all the sessions are recorded. So feel free to go and check out our recordings sometime on YouTube. In addition to that, I highly encourage you to sign up for the Experience League. If you haven’t already, it’s a great forum where you can post questions or help other people with their questions. And it’s just a great place to learn some new skills and see what other people are doing within the analytics space. And of course, you can reach out to me on LinkedIn. Here’s my link. And now look forward to seeing you. So let’s get started. In today’s session, we are actually going to be going over a few things. First thing is a first tip, improved new and return visitor segments, followed by a second tip, smarter alerts. And then we’re going to follow it up with a Q&A session. So let’s get started. Tip one, improved new and return visitor segments. And so what do I mean by that? How do we compare new and return users without overlaps within a specific timeframe? And to understand why this is important, basically, you know, you have to think about how you’re segmenting your users. So in this session, we’re going to go over a few things. Here’s what you’re going to learn. We’re going to use some advanced segmentation techniques. So including a deeper understanding of visitor-based segments, date ranges to create specificity, exclusions to prevent overlapping data. And we’re even going to leverage Adobe’s new relative date range feature to work with this solution. So let’s start with what we have today. Our current options, you’ve probably seen these. The first one is Adobe has provided out-of-the-box segments for first time visits and return visits. But if you’ve used these, you’re also probably aware that these are both all based on visit number. And visit number is an all-time metric. So let’s say you’re looking in the last month. Well, a visitor’s first time visit may have been six months ago. It may have been two years ago. So if you’re trying to look in a 30-day date range, looking at visit number equals one isn’t going to get you your visits within that month. It’s looking at an all-time. So this is a great tool, but maybe it’s not what we need for looking at how our users are coming back within more recent times. The other option that we have at our disposable is return frequency. And return frequency is great, but there are some drawbacks to it as well. The first is it’s bucketed into groupings that may not meet your needs. These are predefined buckets, and maybe you need longer than one month, or maybe you need somewhere in between seven and 14 days. So having them bucketed like this kind of locks you in place. The second thing is that this does not include any first time ever users. In order to be included in a return frequency bucket, you must have returned to the site. So now you’re only looking at a subset of your total users. And of course, the third one, and I’m sure many of you have actually had this problem, if you sum up all of the values, you’ll notice that they actually add up to more than your total uniques. And that’s because your unique visitors are actually going to show up in multiples of those buckets. So a user that comes in today, comes in two days from now comes in eight days from now comes in two weeks later, it’s going to show up in multiple buckets. And that’s going to add up when you’re trying to do a comparison. And if you’re trying to do a donut chart of separating out your users, again, you’re representing more than 100%. So we’re going to create a repeat visitor segment. And this is actually going to be pretty simple. We’re going to create a definition, and we’re going to use days since last visit is less than or equal to the timeframe that you are looking at. So in this case, I’m looking at 30 days. The other thing that I’m going to do is I’m going to pair this with a specific timeframe. So I’m looking for days since last visit is less than or equal to 30, but also within the last 30 full days. And this is going to be more important when we get to the next segment. We’re going to start with what you might think is the segment. And that’s why I’m calling it an interim version because there’s a detail here that we’re missing. So if we look at this, it’s actually based similarly to what I did on the repeat visitors, except now I’m using is greater than and again, 30 days. But I’m also pairing this with an or statement for visit number equals one. So now I’m looking at users who have their first time ever visit or haven’t returned in more than 30 days. So that’s going to bring both of those user groups that we couldn’t get in our return frequency into our group. And again, I’m still going to pair this with last 30 full days. And remember what I said about last 30 full days being important? Well, again, this is going to make sure that your visit number one isn’t an all time visit number one, it’s visit number one within 30 days. So that’s why it’s very important. So if we take a look at a user profile, let’s say we have a user that comes in on January 1, comes in again on February 3, and then comes in again on February 8. What we have here, you’re actually seeing is the first visit is greater than 30 days. And then the next one is within six days. And what happens here is that without any additional conditions, that interim version that we just created, that user is going to show up in both buckets, just like your return frequency, because it is matched true for greater than 30 days. And it’s also matched true for less than 30 days, because of how we’ve split this up. And what that’s going to look like, if we were to pull this into a table, here’s all unique visitors, which we’re going to use as a counter to make sure that we’re seeing how things break out. And then I put my two segments side by side. And we do see values in each of these columns. But what you will see here is that when you add them together, the sum is higher than our total unique visitors. So obviously, that’s not good. This actually results in almost 24k double counted unique visitors in this scenario. So let’s go back to that interim version, and actually turn this into a final version. So I’ve just pulled up the segment as we built before. But now I’m actually adding an additional condition of an exclusion. So I am excluding days since last visit is less than or equal to 30. So we still have our condition for greater than 30 or equals the first visit, it’s all within 30 days. And then it’s going to exclude anyone who also happened to come in less than 30 days. And what this is going to do is it’s going to put all of those return visitors into the return visitors bucket and take them out of new visitors. And you’ll notice again, it’s just just correlating the data is greater than and is less than or equal to, they just match up. So whatever number you’re using, make sure that those two correlate to one another. Now let’s visualize what this means. We have our repeat visitors bucket, and we’re new visitors bucket. And then of course, we have that segment in the middle, where we have visitors who are both new and return. And adding that exclusion, in my case, when I’m excluding it from the new visitors is actually going to take that middle segment and force it all into repeat visitors. But maybe that doesn’t work for you. Maybe you actually want to look at people who are coming back, people who have come back who are new visitors or have returned, you want to see exactly what they’re doing. You just put the exclusion into your repeat visitors bucket instead of your new visitors bucket. And now all your data is going to show up in new visitors. And nothing stopping you from creating two different variants of these segments. So you can actually have both because it’s all custom work. So let’s go back and take a look at this now, again, side by side, we still have our unique visitors. And now we’re going to use our two segments side by side with the final version. And again, we have values in each column. But when we add those columns, the sum now matches. So proving without a doubt, that we have no repeats, no double counted users in this scenario. So we now finally have isolated users. So you can customize your date range. You know, once you have these basics down, you can use this technique for any timeframe, you can use some of the pre built ones that Adobe provides you seven days, six months, one year, or you can even create any custom date range that you need by using the advanced date range selector. So again, you can see here we have some rolling dates, and I’m looking at an eight week range, have fun, find something that works for your business and go with it. Now let’s take this a step further. Some of you may or may not have seen this feature. But Adobe has just rolled out a make date range components relative to panel calendar. And that is available within your panels date range selection. Now before that solution came about, I always had to do a big note in my reports, telling people that the segment was locked to last 30 days, and that changing the report date would not work for that panel. And what you can see here is this report is showing February 3 to March 4. But if I were to change it to December 1 to December 31, none of my values have actually changed. They’re all the same. And so if we were to apply make date range component to the relative panel, now our values actually do change because they are relative instead of locked in. However, there’s a little caveat that you might want to watch out for. And that’s what I’ve noticed is when you have a date range selected, like last 30 days, the relative date range actually starts from the first day of that selection, not the whole panel. So if I were to do last 30 days in December, I’d actually be pulling numbers for November. So I made a little modification. Instead of using last 30 days, I said this month. And then I use a relative date range, which again is relative to this selected month. Now I just put in a note saying that this segment looks at date relative to the selected month, and it’s recommended to look at a full month of data in the report. So again, just minor little tweaks and differences depending on how you’re planning on using it. But now everyone knows exactly how it works and can make use of it. So let’s do a summary. First, a challenge, we needed to be able to segment new and return visitors with better granularity, and to see both recent usage of the site and without overlaps. And our solution, using some advanced segment techniques, we can customize everything to fit our business needs. And now we can understand our visitor groups. So the result, non-overlapping segments, we can have multiples, and we can use them in any of our reports with no problem. So let’s move on to the second tip, smarter alerts. So many of you have probably seen this trend. A lot of companies have moved to a global suite implementation where we have multiple websites and multiple apps and maybe even third-party components for those sites all being tracked in one global suite. Now, a lot of our alerts are still based on kind of that one site notation, that idea that we want to look at totals instead of breaking it down. But as we get more complex, we need to also make sure that our alerts are functioning better for us so that we know where things are happening. The other thing that we want with our alerts is to always reduce false positives. So we’re going to look at a whole bunch of tips in relation to that. So what are we going to learn today? Again, we’re going to use some advanced segmentation techniques, which include isolating sites and platforms, creating time specificity using standard dimensions. We’re also going to use some calculated metric techniques, such as rate calculations, using segments within your calculated metrics, applying functions to create additional logic. And then we’re also going to even take a further look at some external tools to help support your alert management. So let’s dive in. First thing that we’re going to look at is consider using calculated rate metrics for your alerts rather than raw numbers. And why would we do this? Well, calculated rates are going to be a lot more stable than a raw number. So here’s a few samples of some calculated rates, but you don’t have to limit yourself to these. Find anything that you can create a rate on. I’ve got one which is a crash rate, so crashes over launches, or even just a standard page views per visit. And these are going to be a little bit like, well, more stable than what you’re going to have in just looking at page views on its own or crashes on its own. And when I create a site platform segment, I’m going to actually break this down. So you can see here I’ve got a mobile app for iOS. So in my case, I’m using responsive site breakpoint is equal app, or it has an app ID exists. The data is going to look different in your system. It all depends on how you’re tracking. So you can’t take this one specifically for your own needs, but you can look at how you’re tracking it and how you can identify your apps specifically. And then I’m going to also create maybe segments for external site content equals classifieds, or maybe I’m just looking for a specific server domain.com. All of these are different ways of separating out your platforms and your sites from one another so that you can use these into your calculated metrics and your alerts. So external site content and server. Now, when we create our alert, I’m going to use that rate metric that I created before my crash rate. And I’m going to, you know, put a threshold on about 5%. And now I’m going to look specifically for my iOS mobile app, I will also create a separate alert for my Android. What this does is it allows me to see which specific app is crashing, is it iOS or Android, and make sure that when we get alerts, the right team is looking at the right app. And remember how I said that rates are more stable? Well, here’s a sample of why that is. If you look at our crashes and our launches throughout the day, you know, you see, it’s it’s widely varying from off hours, it’s going to be low to on hours, it’s going to be much higher, but the crash rate is stable throughout all that time. And it’s not just through the day that you need to worry about. It’s day of the month, it’s month of the year, depending on your business, you may have, you know, sales that happen at certain times of the year, which is going to have much higher traffic. But still, your rate should be fairly stable. So again, this is going to help churn out and reduce some of those false positives, because you’re not looking at a hard locked number of crashes, you’re looking at a rate. So the next thing we want to do target specific sites and platforms within the architecture. So again, if I have multiple mobile apps going into my global suite, I’m not just going to create an iOS mobile app, I’m going to look at my iOS mobile app for site A, or I’m going to look at, in the case of where I can’t use a rate, I can still use my standard metrics page views is below or equals to 1000, specifically for site A, or site B, or companion site that belongs to site A. For example, we have classified site, which is handled by a third party. If they push something that actually breaks tracking on our classifieds, I don’t want my internal team going off and trying to investigate why our page views dropped when it was actually a vendor and a vendor site that has the problem. So by targeting specifically to our core site, to our external site, to our app, that’s going to give me the flexibility and the visibility into where the issue is so that the right team can focus on something. And of course, if they are properly named, everyone who gets the alerts should know exactly where it is as well. So I am calling it specifically site A Android app. And now everyone knows exactly where that alert is happening. The other thing that you might want to look at is controlling when your alerts are sent. And so if you look at this, I’m going to actually create a calculated metric or sorry, a segment, which actually looks at the hours of the day. So I’m creating this hours of the day is greater than or equal to six, an hour of the day is less than or equal to 10, which remember, it’s hour of the day, which means 10 to 11. So this actually is looking at between six and 11pm. But that’s when I want my alerts to be sent. Now I’m going to use that segment inside my calculated metric. So you can see right there, and I am actually going to pair it with the most inclusive metric. And you might be wondering why. Well, again, if I have any data that happens within those time frames, I want this alert to be active. So I’m going to use the most inclusive, because I’m not actually going to report on occurrences, because what you’re going to see is, we’re going to use we’re going to use a value if true. So the value if true, in this case, is actually going to be the metric that I’m checking. So when I’m between those times, and there’s any data, I’m going to populate out orders. If I am not between those times, I’m actually going to hard code a value, which isn’t going to trigger an alert. So in this case, 1001, it could be any number, it doesn’t really matter, just so long as it doesn’t hit your threshold. So when I create my alert, I’m going to use that custom threshold orders. And I’m going to say alert when it’s below 1000. And if we were to pull this out into a table, this is exactly what you’re going to see, I’ve got my custom metric, and times of the day. And you can see right up until 6am, everything has been overwritten with that fake data of 1001. So I’m never going to trigger alerts in those off hours in the middle of the night. But as soon as it becomes 6am, that real data starts flowing in. And now the alerts can technically be triggered if it falls below 1000. And this doesn’t have to necessarily be, you know, for your time zone, if you work in an international company, you might want to create multiple alerts for different teams. Maybe your North American team has one set of alerts. Maybe your European team has a different set of alerts, play with it, make something that works for you. Now, with all of these extra alerts that you’re creating, you’re probably wondering, how on earth am I going to manage that? Well, here’s that where I talked about using third party tools to help you out. If you have Slack or Teams, and maybe there’s some other tools that your business has that has this capability, there is actually an email integration. So you can get an email address, which will send directly to those channels. And if you use this, instead of having individuals sent to all of your alerts, you send all of your alerts to a shared channel. And then everyone who needs alerts can be part of that channel. They can manage their pause notifications, they can manage what times of the day they want to be alerted, they can pause it during meetings. If they’re on vacation, they can pause it, they don’t have to take themselves off the list, put themselves back on. And the beauty of this is you can see everyone who has signed up for alerts. So if somebody joins the team, or somebody leaves the team, it’s one click to add or remove a user, as opposed to going through your 20 billion alerts that you have set up. And everyone in this channel can now go and look at the alerts and see exactly what’s happening. So in summary, as our infrastructure and implementations get more complex, we need to make our alerts smarter to be more flexible and more efficient within those infrastructures. So our solution, using a combination of calculated metrics, segments, third party tools, we can build an architecture that works for us. It is tailored, which is the key. You want to have tailored alerts, because if everyone’s getting alerts all the time, they’re going to stop listening to them, and they’re going to start ignoring them, and then no one’s going to actually pay attention. And the result? Smarter alerts will make things faster and easier for your team to track down issues, which is the key of having alerts in the first place. So everyone here, you’re all smart people, you can use these tools Find things that work for you. Take these tips, expand on them. I’m sure you’re going to come up with some amazing ideas to make your alerts better. And hopefully you’ll be sharing them in a skill exchange or another event, just like me. Thank you. Jen, that was wonderful. Thank you for sharing and joining us today. How are you? I’m good. Thank you for having me. Good, good. We have got the questions coming in our chat window. Let me pick the first one. So we have a question around how confident are you in the return visitor segments given ITP and cookie deletion? Well, also, like, can you explain what ITP is to anybody in the audience who may not know? Sure. So for those of you who aren’t aware, ITP is Safari’s intelligent tracking prevention. It started with them deleting third party cookies and moving forward and starting to delete first party cookies much quicker than normal. I think first party cookies get deleted within seven days now. And of course, that always causes a concern with our analytics and our return visitors because those visitors no longer have the cookies to identify them. So you’re right to be concerned. And I obviously have concerns myself. One of the ways in which that we can actually kind of move forward from that is looking at the Web SDK because Adobe’s Web SDK is actually going to use server-side cookies instead of client-side cookies. So the visitor segments should be much more secure when moving to that new structure. Right now, of course, anyone who’s still using traditional, like myself, definitely has a problem. And one solution that my organization came up with was to make copies of all of the client-side tracking cookies, store them as server-side cookies so in the event that they got deleted by ITP or other cookie deletion softwares, to put them back on the server with the values. That’s not an option for everyone, obviously. So there’s always ways that we can try and get around it, but there’s always going to be potential problems with identifying people. Of course, there’s always people who use Incognito, who delete their cookies manually when they close their browser. So is it 100%? No. Will it ever be 100%? Probably not. However, looking at trends, it’s still close enough to try and get a feel and a sense of where your visitors are sitting and identifying them. And that’s the best we can do at this point. Okay, that was insightful. We have the second question. That’s something that I wanted to ask. Can you actually create a segment to send alerts between specific hours of the day and say like only on the weekdays so that alerts won’t bother people in the weekend? Oh, absolutely. So if you remember in my presentation, I was showing the hours of the day constraint by using a segment. Adobe has a standard dimension called weekday and weekend, or weekend. And basically, you can use that to identify your weekdays and weekends. So within your segment, you just keep adding on and clauses to look for specifically weekdays. And in doing so, you can get even more creative. You can build in lots of different logic towards when you actually want the alerts to get sent, and that segment is going to drive most of the logic. So there’s lots of options. Play with it. Just make sure you test it first, because if you do something wrong, that’s going to cause a little bit of a problem for you. And you don’t want your alerts only sending in the middle of the night and on weekends, because that’s not going to help anyone. But you can always test those segments out before you actually put them into your alerts. So bring them into workspace, pull data in by hours and days, and look at everything and actually make sure that the data being returned is the way you expect it to be. And once you’re confident in that data, similar to, again, what I showed where I was showing the fake data and the real data, just make sure that everything is lined up the way you would expect it in a normal scenario. And then you can actually go forward with your alert. Okay. That sounds like an easy process to do, but yes, I will remember that test your segment before you actually put it to use. With that, let me actually come to the next question. This is, can you explain in more detail why you need the last 30 full days constraint in the new and repeat visitor segments? Absolutely. Now, in your repeat visitors and your new visitors, you think technically you shouldn’t need that. But when you add in the logic for visit number equals one, when you’re trying to capture those new visitors, those first time ever new visitors, you actually want to make sure that that is constrained by a reasonable time that matches up to the date range that you’re looking at. And of course, even though you don’t have that new visitor visit number one in the other segment, you want to make sure that both segments are kind of following similar logic. So it’s just kind of a safety procedure to make sure that both segments are kind of processing with the same basic logic. And of course, it doesn’t necessarily have to be 30 days if you’re looking at return visitors or new visitors in six months, then you adjust that time to match up with whatever segment you happen to be using. Okay. Okay. One more question. You know, how in off-peak cars, you know, the traffic has a seasonal and even daily behavior. So we often get a deluge of alerts for drops in traffic, which are genuine as the traffic falls across the device platforms in a day. So how would you go about, you know, normalizing the alerts for dips in the traffic in these off-peak cars? Right. Well, again, that’s one of the things about rate, because when you’re looking at rates, your visits are going to drop proportionally with your page views, are going to drop proportionally with your conversions, are going to drop proportionally with different factors. By creating rates, that should help normalize the data because you’re actually looking at, you know, your data against, you know, all the drops as opposed to just raw numbers. Of course, I know it’s not available for everything and that’s where you end up having problems. But if you get creative, you should be able to say, okay, we know seasonally, we get much more traffic maybe in November during, you know, big Black Friday sales. I know that’s a big thing in the US. I’m not sure about this audience, but surely you know your systematic, where you get your highs in the year, where you get your lows in the year. Now, this is going to be a little complicated, but you can actually create a very complex calculated metric, which actually pulls in a lot of if logic and you can nest that if logic because unfortunately ifs are only two conditions. But if you look for if it’s January, do this. Else, like in the false statement, you pull in another if statement. If it’s February, do this. Then the false, next one, pull in if it’s March, do this. So you can start to build very, very complex logic in this if structure so that you’re actually looking at different values against different months. And that should help alleviate some of those traffic falls or traffic growths based on seasonality. Okay. Okay. Do you know like just deep diving into your previous question. And again, this is the last question we have because we are running out of time. How do the alerts compare the data, you know, between this week and the previous week? Right. So alerts aren’t really comparing it on a weekly basis. It’s really just comparing it against whatever you’re checking, the now. So if you’re sending out a weekly alert, it’s what happened this week? Did it trigger? Did it not? What happened in this hour? Did it reach that threshold or did it not? There isn’t really a comparative to past data unless you’re building that into your calculated metric where you’re actually doing comparisons against previous data. And again, we can go down a rabbit hole of logic here, which without actually having a computer screen in front of me and sharing my screen about how to dig into this, this might be something that if you’re really interested in following up on going to experience league, because I’m there every day. We can share screenshots, look at logic, try to build out complex calculations to try and actually see what exactly you’re looking for and try to get you solved for your issue. All right. Jennifer, thanks again for joining us. Those were some brilliant insights into some of these questions. Thanks a lot. You’re welcome and hope to see you in experience league. See you.