Scaling Personalization and A/B testing at TELUS
Data Intelligence · Mar 25, 2019
TELUS brought a team of technically savvy people together to unlock new personalization capabilities and scale the practice across an organization in an automated fashion. Resulting in hundreds of hours saved in running a/b tests, increasing user experience and performance.
Historically, TELUS personalization team had done an amazing job running A/B tests with the out of the box Adobe Target solution. However, our performance was suffering and customer experience was being impacted which called for a deeper dive into understanding the issue.
With TELUS experiencing an increased emphasis on performance, the Analytics & Insights team was also going through a transformative phase. The tools we were using weren’t necessarily optimized for speed. Throughout the year, we implemented brand new tools such as Adobe Launch and Adobe Target Server Side with performance improvement as our main goal.
The following post follows the story of our personalization practice and how we saved hundreds of hours and scaled the personalization practice. We developed a solution that scaled on technology, not people, while enhancing the customer experience and improving site performance.
Within TELUS, Content Managers were using Contentful to build content, and our personalization team was using Adobe Target to deliver personalized experiences with Adobe Target. One of the first issues we noticed was a “content flicker” with this approach. At the time Adobe Target had not released a working version with React. Moreover, we were at a crossroads on how to provide a solution that would meet our requirements.
To better serve our customers, TELUS was making significant investments in personalizing content, offers and experiences. As a result, we needed to find a middle ground between Adobe Target and our ReactJS frontend.
"We wanted to provide a way to marry the power of our digital analytics stack while at the same time providing a single interface where our content practitioners could create and monitor the performance of their a/b tests & targeting use cases." - Dinesh Mahtani, Director of Analytics
After working closely with our Adobe consultants and our talented platform team, we ended up with critical requirements for the next phase of evolution called the Personalization API:
Clear separation of concerns between systems
Contentful should remain the only source for storing and retrieving content.
Adobe Target should be the single decision-making engine based on parameters such as Adobe Analytics Data, and Adobe Audience Manager Segments that both include offline data
Any API developed should respond in less than 200ms
Improve client-side applications performance (Lighthouse score)
Empower product owners to increase conversions by simplifying Personalization & A/B testing workflows
Our solution moved the sequence of events of personalization from the client to the server. Essentially, it would allow us to remove the flicker, increase our response times, and improve our reporting.
The Personalization API is middleware, and also acts as an aggregator between systems. The sequence of events is as follows:
Client application first requests content from Contentful
Our Personalization API intercepts the request and calls Adobe Target to find the correct variation to show
The Personalization API stitches up together the personalization content into the original page
The client application receives a personalized experience straight from the server
While moving the sequence of events from the client to the server, we were able to meet all our requirements and have an average response time of 140 ms.
This solution has allowed us to scale the personalization and A/B testing practice throughout TELUS as well as improving performance and customer experience. Overall, it has been a hugely successful product launch and, our engineering team, Fernando Alfaro, Ozan Coskun, Ajay Ajaal, Gayan Pathirana, Attiq Ahmed and myself have pushed the envelope of what personalization can become and will become in the future.
Fernando Alfaro, our Senior Data Architect, is presenting the solution at the Adobe Summitnext week and our Adobe friend Ahmed Elemam, has been helping us with the adoption of this approach across all our outcomes teams and connecting us to technical minds inside Adobe.
It takes a lot of foresight to see the possibilities in an ever-changing data-driven landscape, the investments we made were heavily back by Dinesh Mahtani who saw the potential and propelled the organization forward.
“The end is a mirage: excellence is a process, not a destination” - Brian Tracy
Our first release was to add a wall to our mobile web experience to encourage customers to download our mobile application MyTELUS.
The Personalization API release was a great success since we saw a 3% conversion rate from end to end experience. Our API response time was less than 150 ms, and metrics were available almost instantly in Adobe Analytics.
As we were getting ready to celebrate, we got invited to participate in a hackathon in March organized by Andrew Douglas (Manager, TELUS Design System & Content Platform). The event was meant to foster collaboration between Content Practitioners and Technologists at TELUS and Contentful.
The prizes were incredible, first prize being a flight to Berlin to visit the Contentful headquarters.
Having just released a solution that improved the user experience of our personalization team, we decided to take this opportunity to focus on the user experience of the Contentful practitioner when running personalization and A/B tests.
As the hackathon began, we pitched the idea of surfacing analytics data straight in Contentful. The latter would allow content practitioners to choose winning experiences sooner and democratize data to empower them to make data-driven decisions.
Our team was made up of four developers (Ozan Coskun, Attiq Ahmed, Tim and myself, Gonzalo Vazquez) and two content managers (Anne Booth, Tiffany Moreside). We started by addressing the pain points that content practitioners experience when running Personalization and A/B testing:
Impossible to see in Contentful if a page has personalization.
It takes an average of twenty steps for a Content Practitioner to find if a page has personalization and find all the variations.
Analytics are not easily accessible and are restricted to only certain individuals in a team with a specific skill set and training.
Content Practitioners do not have access to Adobe Analytics and Adobe Target.
Content Practitioners have to set up meetings with Analytics to see how well a variation is performing.
“Innovation distinguishes between a leader and a follower” - Steve Jobs
Our team spent the first couple of hours setting the context for Adobe Target and getting familiar with the personalization practice. Afterwards, we took our ideas to the whiteboard and brainstormed on what information we needed to show to improve the Content Practitioners experience. We decided that unique visitors, conversion rate and traffic were key indicators to provide the best user experience.
After several hours of coding, designing and many iterations we had finished our UI extension named: Beyond Insights
By surfacing Adobe Target and Adobe Analytics data we were able to build a UI extension in which Content Practitioners could, at a glance, see the winning variations with real-time analytics.
The contentful integration our team built would have resulted in almost one million dollars in savings back in 2018, and we look forward to seeing how much this extension will save us in 2019.
However more importantly than savings, this Contentful extension is going to enable us to continuously have contextual data on demand, and available in a convenient place for all users regardless of access or domain. Overall it was a great experience and I feel lucky to be part of such an incredibly talented team and organization like TELUS.
It was going to be a hard decision for the judges to choose a winning team since all the teams were incredibly strong and the demos well elaborated. Since our team was presenting at the end of the session, we were getting worried if we had a good enough idea to impress the judges.
Finally, the deliberations happened, and the first place was awarded to the “Beyond Insights” team.
Undoubtedly, We were all in a state of shock after winning, words could not express what was going through my head. Having a great idea is part of a solution but having a good team that respects and plays to each other strengths was the winning recipe.
Our hackathon team will be flying to Berlin for a week during the summer to make this extension a reality at Contentful headquarters.
Thank you, team, and here is to many more!
"Talent wins games, but teamwork and intelligence win championships." – Michael Jordan
Do you think you have what it takes to be on #TeamTELUS? Apply today and reference this post in your application. Click here to apply!
