Navigating SPA Challenges in Test Builds & Analysis

Transcript

Welcome to Navigating Single Page App Challenges and Test Builds and Analysis. I’m Kathy Salek, and what I’m going to share with you is what I learned going from a multi-page test environment to a single page test environment when I joined Panera Bread. So I’m going to start by telling you a little bit more about Panera. Panera is a leading restaurant brand with over 3,200 bakery Catholic cafe locations in the United States and Canada and annual sales in the billions. You might be surprised to know that our e-commerce is over 45% of sales. We have high traffic and conversion, which makes us an excellent candidate for A-B testing. In this presentation, we will cover differences in single page app and multi-page applications, building an AT.js 2x, test entry and impression counts, in-session test qualification, and a refresher on inbox parameters as well as analysis considerations. We’ll start with site architecture. So one of the most important things to remember with a single page app is that your refresh only occurs once. There’s never an additional refresh. Within that first page load, all HTML, CSS, and JavaScript comes down to the browser. It’s much more like a native app in that way. With a multi-page environment, which is a traditional website build, those files come down on each page load. So as someone navigates, the HTML, CSS, and JavaScript for each page is coming down to the browser from the server. So there are lots of opportunities to reload. At Panera Bread, our main website is a single page app, PaneraBread.com, as well as catering.panerabread.com, and our native apps in iOS and Android. Let’s jump right into builds. So before you build, one of the most important things to do is just make sure that your trigger views are set up correctly. The way that you do that, the easiest way, is to go in into an activity, and within the Visual Experience Composer, click on browse. When you click on browse, you can navigate through your website within the Visual Experience Composer. You’ll want to watch your current view to make sure that it is updating. If you’re not exactly sure what it should be updating to, you can always follow along on your website using the debugger and checking your analytics calls. But typically, you’ll have some idea what those views should change to, and you can do this without that. Best practice is to leverage a page name variable when you’re setting up your trigger views. That’s from the data layer. This should be used for analytics, like analytic page names, as well as your Adobe Target view change information for trigger views. At Panera Bread, our data layer is a little slow, so we’re not able to set it up that way. The way that we do it is with the path name URL. So that changes in most areas of our website, and we’re able to use that. That’s a nice alternative. Page delivery is probably the most important part of your build. For me, it was the biggest change. In AT.js 1x, there’s not really a concept of views, and so your page delivery is a lot more important in determining where your test is actually going to deliver. Also in AT.js 2x, your page delivery determines where your content is prefetched, because remember that the single page app is only bringing all of its contents down from the server once. So we’re prefetching. Prefetching is a technique used to boost execution performance by fetching what is needed from the server and storing it before it is needed, hence prefetching. It stores it in the cache. The cache is a storage space for temporary files that makes a device, browser, or app run faster and more efficiently. Best practice is to set your page delivery rule across your entire site. The reason is that no matter where someone enters, they can get into your test activity. You’re not yet delivering the content, but you’re delivering the payload or the information that’s needed for them to get the test experience. So you want to cast that net as wide as you can so that when they do reach the appropriate point, it’s there. We’re going to talk a little bit more about modifications. Modifications control where your content is actually delivered. Page delivery controls where it’s fetched from and modifications where it actually delivers to. Your modifications can be found in the Visual Experience Composer. Under compose, you’ll see add modification. It’s the same as in AT.js 1x. But what’s different is page load and view in AT.js 2x. So page load should be used for global modifications. It will bring your contents into the head and they will persist on all views. Your views, if you set up your modifications to apply on view, will only apply to the specific views that you add to your modifications. I included this because it didn’t really occur to me that you could have multiple views with different names. So actually it’s the same view with multiple names. So in our case, our menu is really important and we have our menu listed across multiple views. There’s a reason for the differences. In some cases, they function a little bit differently, but for the most part, it’s the same content. So when I run a menu test, I want to make sure that I’ve included all of these menu variations or different views because it’s essentially the same element. Where your content is applied determines your test entry and impression. So we had mentioned that for page load, we want it to apply that for things that we want global modifications for. If your page delivery rule is set to the entire site and you choose page load for where you are delivering your modification, you’re going to have everyone who meets your audience qualification that visits the website in your test. So that’s something to take into consideration. If that’s not your intent, if it’s not a global change, then you don’t want to use page load. You want to use view. View will give you more of a surgical way of deciding at what points people are getting into your test. So if your test is only built on the homepage, just use the homepage view for when and where to deliver that content to only get people who are visiting the homepage into your test, assuming you have an all visitors audience. Native apps are a little different. They require some additional work in their setup and the instructions I have here may or may not apply to you depending on your native app setup. This applies to us. And so I thought I would share it. It was something that I learned in setting up our native app implementation. Ideally when your app opens, it fires a call to prefetch content and then retrieve content, retrieve location content. When used in this order, you will not increment an impression. Target will send its contents, whatever your test is as a string, which can be XML, JSON, or just plain text. And when the user reaches the test location, the content will apply and a locations displayed should fire. If your native app skips prefetch content and goes straight to retrieve location content, kind of like as a substitute for prefetching, you will increment an impression at that point, which means that you can count people when they open your app in your test activity. It’s important to fire that locations displayed and the prefetch and retrieve location content in the order that’s laid out here to keep your impression count accurate. Audience qualification. So ideally you want to be able to qualify people into your activities as they’re doing things on your website. To do this, you will need to refresh your prefetch cache. In the example I have here, when I enter the website, I’m not yet logged in. So it shows that value is false. When I do log in, my value changes to true. When I select a cafe, my local cafe ID that I selected shows up here. And the way that you will set this up is by using get and apply offers in a launch rule. You want to make sure to keep your page value false. This will prevent additional impressions from being incremented when this is called. The get offer function will fire a request to Adobe target that will get that information in the case of the example that I’m using here and we’ll apply it or process it into the inbox. The apply offer will process it. The additional data needed for the experience qualification will be passed as inbox parameters. Data that is needed across sessions should be passed to Adobe target with a profile dot prefix. This will be stored in the Adobe target visitor profile. That profile can live for up to 90 days. However, you’ll have to email Adobe customer care to have it extended. By default, it’s 30 days. All in session only or non-persistent data should be passed without a prefix. You can collect data in both ways. For example, I collect cafe ID as a profile dot parameter, as well as a standard no prefix parameter. And the way that they’re used is when someone comes to the website and I want to customize early in the funnel like the homepage and the precision is not as important, I can use something from a prior session that visitor profile data that we’ve previously collected about their cafe preference. However, there may be instances where we are customizing further in funnel and may want to be more precise. Someone may choose a different cafe depending on whether they are ordering from home or office or traveling. And if that precision is really important, then I want to wait until they are in session and tell me specifically what cafe they’re ordering from and then alter the flow downstream of that. When you go to use the data that you’ve collected to build your audience, you will go to the audience tab in Adobe target, just like you normally would to build an audience. And when you select new audience and you go down to add rule, how you’ll access the data is the data that’s collected without a parameter. The more standard original parameter will be under custom. So you’ll go there to start building your audience. And then for the data that’s collected with profile dot, you will find that under visitor profile. Now onto analysis considerations. In putting together this presentation, I had pulled in a lot of stuff and then I realized I could have a whole separate presentation just on that. So what I’m hitting on here are some of the things that I thought were more important and still stayed true to the the rest of the content in the deck as some learnings that are important for single page app test builds and analysis. I’ll start with the obvious. Your A for T panels work the same regardless of what Adobe target library version you’re using. I do have a recommendation here. There’s not a ton of space to look for your test. And so I would recommend using something that allows you to quickly differentiate your tests. One thing that we do is we use our JIRA ticket numbers and it allows us to tie things together across JIRA and Adobe target. And then here in analytics, this kind of goes back to the trigger view. This is just something that you want to check if you’re starting to test in a new environment, you want to make sure that your hits are stitching correctly. So for every Adobe target SDID, the supplemental data ID, you should have a corresponding analytics SDID. For every target call that delivers content or records a goal metric has this SDID and the corresponding analytics ID. So it keeps everything tied together nicely in your data. Make sure your calls are in the right order. It’s very unusual that they would be in the wrong order, but it is possible. Your target calls should be first and they typically will be because target needs to load very quickly to apply all the content before your visitor realizes that you’re changing the content on them. And your analytics tag typically will come in at the bottom of the page load or the view refresh. If that is not the case though, it can cause stitching issues. And so that’s why I mention it here. In our case, we ended up needing to chain two separate rules together to keep them in the right order because they were separate rules that were not aware of each other. Segments I think are particularly important in AT.js 2x. They’re important in 1x, but in 2x, they can help you detect issues in your build. If you start applying segments to really get into just the people who were in the right place for your test experience and you see the numbers changing quite a bit, then you know that you may have something to look at. You may have set the test up as a page load instead of a view, and you may have a lot of people in there who didn’t actually reach the area of your test, and so you want to filter them out. You can also narrow your results to just the sessions of exposure. So for example, if you were running a PM test, you’re still going to get people who were cookied into the PM test but came in the AM. Like if I come, I buy pizza in the evening, I’m cookied into the test, and then the next morning I buy breakfast. I’m not getting the experience, but I’m being reported in the analytics, so you can filter that out or choose to use that information for a separate analysis. Maybe you want to look at a residual impact. For example, if you have a weekend offer and you want to see if it’s driving any additional weekday traffic or purchases or what it’s doing to long-term purchase history, you can look at that additional data. Just be cognizant of what you’re looking at. Monitoring distributions across key segments can help you find distribution issues or delivery issues. Shifts in your distribution should really only occur after the point of the test activity. For example, if I have a test where I’m promoting pizza in my test experience, I can probably expect to see people who are visiting more of the pizza pages and purchasing more so in that category within the test group. That’s a skew that I might expect between test and control. On your entry pages though, you really shouldn’t have any kind of distribution issue. In this example that I have listed here, it shows how a build can impact your distribution. Let’s say I have an activity that I’ve built and the page delivery rule is for the entire site, but my control delivery rule is on page load. Normally you might not have code there, but in this example, let’s say I have a console log that I’m using for other purposes. I do have some code in my control. It’s just not impacting anything on the view or the page. Then in my test experience, I am loading the code by a view modification. What can happen there is that you may see more people in the control group that have your overall websites most popular entry page as their top entry page. Whereas in your test, you may see that it’s a different entry page and it’s an entry page that is popular among people who visit that view or views wherever you’ve made your changes. That’s one example of how you could have a distribution issue that could impact the results of your test. Depending on how big of an issue that is determines whether or not you restart your test. I would probably restart it on a distribution issue like that. In summary, you will set up page delivery across the site to ensure your content is fetched regardless of where someone enters on the website. Use view modifications unless your change is global, then use page load. Refresh the prefetched cache at key points such as login, add to cart. It’ll depend on what your business is and what data you need to support it. That’ll allow for in-session test qualification with your audiences. Use profile.prefix to store longer term data. When you use that, it can live for up to 90 days if you have your visitor profile extended. And no prefix for those standard parameters that are in session or non-persistent. Filter your results to match your test delivery and or objective. Monitor distributions to detect build and stitching issues. And last but not least, I want to give a special shout out and thank you to my team at Panera as well as our strategic partners. Without them, this presentation would not be possible. I have learned a ton from each and every person on this slide, as well as many more that are not included here who have just made what we do possible. I hope that you find value in this presentation and I really appreciate being invited to participate. Thank you for having me. All right, thank you so much, Kathy. It’s great to have you. Thank you for joining me today in front of what I’ve termed the Skill Exchange fireplace, which is patent pending. So, Kathy, thanks for joining me. Thank you so much for having me. I’m thrilled to be here. Yeah, yeah, I loved your presentation. I love the focus on all the kind of crazy, weird things that exist within single page applications and trying to optimize for them within Adobe Target and analyzing all of that data, all of that good stuff. And we’ve got a wealth of really great questions coming in from the audience. As a reminder, if you have questions for us, don’t hesitate to throw them in the chat pod. And if it’s a follow-up to one of the answers that Kathy provides, then that’s perfect. If it’s completely unrelated to what we’re talking about but still is related to Adobe, then don’t hesitate to ask that question as well. All right, so for our very first question coming from an Adobe Analytics champion herself, Sara Owen. This is a really tough question, Kathy, so get ready. Are you ready? I’m ready. Okay, all right. So, Sara is going to want to know what A-B test she’s currently in whenever she’s ordering an autumn squash soup. So, what kind of recommendations do you have for Sara there in terms of her soup ordering? Well, Sara, there’s a couple of ways to figure that out. One, I like the developers console. You can go straight into the developers console as you’re navigating the site. Go to the network tab. I search for delivery to find the Adobe Target inbox. There’s other things you can put in there, but delivery pretty consistently brings back your inbox. Then once you’re there, you can unpack your inbox and see what activity you’re in. An easier way to do that is probably to use the Adobe Target, or actually it’s just the Adobe Experience Cloud Debugger. You can also open that and follow along as you’re navigating the site to see what test experience you’re in just by going to Adobe Target and looking at the calls that are coming in there. Awesome. Love it. I love that we had a joke question and you turned it into a really helpful tip, so that’s great, Kathy. One thing to remember is to use those Experience Cloud Debuggers, you’re going to have to authenticate in order to run that inbox trace, if I remember correctly. Is that right? Yeah. To get all of the information, if you want to look at your profile information, you’ll have to authenticate. And to see what activities you were considered for and qualified for, you’ll add that query string to your URL from the inbox trace. Love it. Yeah, and a little known fact, the Experience Platform Debugger, which is the latest and greatest debugger that we have out there, if you happen to be using that and you happen to be using the Experience Platform Web SDK in order to deploy Adobe Target and Adobe Analytics, etc., then you’ll actually get that target trace right within the platform debugger as well. So, potentially a little known fact for you, if you’re on the path of migrating to the Web SDK or you already have, then check out the Experience Platform Debugger and you can authenticate and run those target traces there. All right, so the next question that we have comes from Deepti, and a really great question as well. Kathy, I think you’ll be ready to rock and roll with this one. The question that Deepti asked is she would like to know if you are running multiple tests on the same page at the same time. And if so, how do you decide who to give credit to if both the tests are running and which one is winning and which one is actually getting the conversion? And then finally, a little bit of a follow-up there is how do you analyze those multiple tests if you are running them on the same page? Of course, we’re talking a single page application, so the concept of same page is kind of crazy here. So, walk us through a little bit around how you’re managing and staying organized of these multiple tests within a single page application and even sometimes on the same page. Multiple page tests are tricky. So, right now we don’t really do… We have a few that are multi-page or not multi-page tests. But where you have multiple tests that are running simultaneously with the same audience or same page. Most of what we do is swim-lained. There is a script that you can find, I believe it’s on the Adobe website, in one of their help articles that will randomize your audiences. So, as someone comes in, you assign them to a group, they’ll be in one through 10 or you can do more than that. We do 10. And that keeps things pretty clean. So, we only have people who are in one or two lanes for each test. If you do want to run tests simultaneously where you have people who are potentially getting into multiple tests, you’ll need to spend a lot more time looking at the data and really understanding what the impact of that is. So, there are things to consider when you’re running multiple tests on the same page, like you could have things that impact each other on that same page. I think a good rule of thumb is to try not to run it on the same page, especially if they’re kind of in the same area or have similar functionality, because there could be things that are happening there that you’ll have to tease out of the data later and if you’re even able to. I would keep it to one test per page if you want to run multiple tests and allow people to get into multiple tests at the same time or within the conversion funnel also. You don’t necessarily want multiple tests running within your conversion funnel. And in terms of the attribution, if you are running multiple tests at the same time, as people enter one test and are going to the next test, they should be randomly split into the different groupings. So, from one test to the next. And your lift will be incremental. You can confirm that that’s happening correctly by running a series of AAs to make sure that those splits are happening the way that you expect to. So, you have an AA, for example, on your homepage, your category page, and maybe somewhere deeper like a checkout. And then just making sure from one AA test to the next AA test that those splits are happening as they should, where you’re not getting more of one group from one test into another test into like the B experience or something, for example. And if those flows are happening correctly, then I don’t think the attribution is as big a deal. What I did at the last company I worked for is we were very cautious with what we were reporting to finance. Our numbers were big, and we wanted to make sure that we were as conservative as possible in those attributions. So, I did a weighted average of all of those tests. I did have overlapping tests there. I didn’t swim lane like I do now with Panera. But it depends on your website and the traffic that you get and the stability that you see as people flow through into the different test experiences. Yeah, that’s super helpful insight there, Kathy. And you had a long answer, and I think for good reason, because this stuff is complicated between single-page applications, multiple tests, and then multiple segments or audiences that you can apply to them. You need to make sure that you’re not double-dipping in terms of one test is actually causing another test conversion or vice versa. That’s great. Really good insight. Thank you, Kathy. If you have the traffic for it, I do recommend swim lanes, but that’s a high-traffic solution. Yeah. And then can you actually describe a little bit more about how you’re defining those swim lanes? Yeah, so it’s a profile script that we’re using, which runs, and as someone comes to the website, the profile script is assigning someone a lane 1 through 10 assignment, and then we create audiences based on that. So the profile script is feeding those audiences, which in turn are being used in the test. So we’ll pick lanes 1 and 2, for example, for a test. And then as long as their device browser combination is the same, which is not always the case, but it’s pretty accurate. We don’t have a lot of multiple visitations before someone places an order. So for us, it works really well. If you have a longer sales cycle, swim lanes may not work as well for you because you could have people who are accessing in different ways and getting into different lanes. Yeah, you bring up a really good point that ties back to a conversation that Christos and I were having earlier around when you’re doing sequential segmentation, do you use a visitor-based segment container or a visit-based segment container? And Kathy, you just gave the perfect example of the Panera business where a visit-based segment makes so much more sense because the types of actions that users take on the Panera digital properties are going to be really just like, I came to the site, I expect to order my soup or my sandwich, and I want to check out seamlessly and then go pick it up or get it delivered. And I want all of that to happen within a visit. So I am trying to remember the name, whether it was Dylan or maybe Abe earlier that asked that question. That’s a great use case for a visit-based sequential segment where you can say, what is a part of this activity? And then within that visit performs the conversion or a micro-conversion along the way. Awesome. All right. So, excuse me. So the next question that we have, let’s actually go with this question from Sarah Owen again, where she’s hoping that you can maybe talk a little bit more about how you detect distribution issues within your test data. So, Kathy, do you mind talking a little bit about some of those distribution issues and how you’ve been able to detect them over the years? Not at all. So what we’ll do is we have an Adobe Analytics workspace, and it’s a template that we use for all of our test analysis. It contains a lot, and one of the things that it contains is unique visitors. So we will pull out this, you know, we’ll change it so that it’s appropriate for each test and then export it to Excel. And we have a tab in our Excel workbook that is, you know, a partner with this. And what it does is it looks at the distribution. So in that case, we’re using unique visitor. We use visits for most of our test analysis, because to your point earlier, people are converting within the visit, and that’s what we’re most focused on. But for distribution, you want to use unique visitor. And so what we’re looking at is, you know, overall test unique visitors and then test visitors in each segment. Our segments are based on things that are important to our business, as well as some things that are just kind of good segments to have to understand what’s going on in your test. Some of those would be entry page is probably the most important. You want to understand if there are distribution issues there, because that’s the start of the process. And then also looking at things like device browser, you know, screen size resolution, things like that. And then those segments that are important to our business, like we have some customer attribute segments that we bring in so we can understand how different customer groups are responding to a particular test. But in the example of distribution issues, we’re looking at of the percentage of people who are in that particular segment, like an entry page of homepage, what percentage of them are in the test group and what percentage of them are in the control group? And then we do a T test on the difference between those percentages to understand if it’s a significant difference. And then look at we do a calculation of the power on top of that for that confidence to make sure that, you know, if it’s 99% confident, what is the power associated with that based on the numbers that we have there? Is it also 99% and is that difference like a couple hundred people or is it a few thousand or a few hundred thousand? And then when you get into those larger numbers and you see those higher percentages, you know, you have an issue. Gotcha. Yeah. That’s, that’s a lot of statistics and I love hearing it. Are you doing most of those, the T test, the power, where, where are you doing those analysis? Excel. So for our, for our business partners, so our analysis has multiple pieces for our business partners. We have a dashboard and that dashboard is updated within like 24 hours of a test launch so that they can start to see where their test is. And then we do a weekly deep dive where we pick certain tests and really dig in and understand what’s going on with them. And that’s where we would use the Excel spreadsheet to do those multiple calculations. Okay. Yeah, I love it. I love like that, that high level view of like, give me everything that’s going on. And then the, you know, weekly deep dive, that’s a, that’s a really great way to handle it. I feel like we could apply that same mantra to really anything within digital analytics and optimization and personalization advertising is like, get both the macro and the micro view, but you know, there’s limited resources. So you have to focus on, you know, one specific set of content when you’re or test or, or analysis when you’re, when you’re driving your micro set of analysis, that’s great. Okay. So I’ve got a little bit of a softball for you, Cathy, hopefully, hopefully you’ve got this one ready. All right. So this one, uh, there’s a question from Chris and Chris asks for a single page application for a spa. Um, do you recommend, or can you make the inbox the entire, uh, page or do you have a different way of deploying inboxes that you prefer? Um, so I’m not, not quite sure I understand the question. Um, is like by, I think of the inbox as the marketing box that, um, it delivers the payload of all the changes that we’re making to the site and it collects all kinds of information. Um, maybe is he referring to, um, like just wanting to change the entire page, like all the contents of the page if, if that’s right. Yeah. I think, um, you know, there’s kind of two ways of thinking about inboxes, you know, whether it’s wrapping it around just the piece of content that you would like to change. Like a regional or an event inbox. Right. Okay. All right. Yep. I’m tracking now. Um, I guess it depends on your use case. Um, I’m still a fan of regional and event inboxes. I know they’ve kind of gone away years ago. Um, we had them at H and R block and then we went away with them and I kind of missed them. Um, I think they have their place. So in terms of whether or not to, um, I, I think you could certainly, um, it would be a regional inbox and it would be for that particular view. So yeah, I think you can definitely do that. Depends on what your need is. Okay. Perfect. Yeah. I think, uh, that’s that, that really could sum up our session is, um, every, every website, every digital experience is unique. And so you have to take the pros with the cons in terms of ensuring that using the right technology, you’re deploying the technology correctly. You’ve got the analysis prepped all of that great stuff. So, um, awesome. Well, thank you so much, Kathy, for joining me here at the Skill Exchange fireplace. And it was wonderful to chat with you. Thank you so much for taking the time. Yeah. Thank you so much. Thrilled to be here.