Roblox said it is launching thumbnail personalization to help creators attract more users by showing the most relevant thumbnail to each user on the Roblox home page.
While thumbnails are the tiny images that you tap to initiate a game play, it turns out that the image that is on that thumbnail is extremely important.
It turns out that personalizing a thumbnail is a really good way to draw in audiences for a game. Experiences with thumbnail personalization saw an average 8.5% increase in qualified play through rate (qPTR) with some even seeing a 50% increase, said Chen Zheng, senior director for product at Roblox, in an interview with GamesBeat.
“We have very encouraging data. One developer said this is game changer for them,” said Zheng. “Thumbnail personalization is really part of this strategy to help users better evaluate recommendations in front of them and make a more informed decision” about playing the game.
The thumbnails apply on mobile devices, where users can choose which games to play by looking at a page of thumbnails. On top of that, they can also choose to play a game on the PC or console devices in the same way, where visuals will draw in players.
How it works
To get the most out of thumbnail testing, developers need to keep multiple thumbnails active, instead of picking just one winner to let personalization adapt to changing user trends.
This is because Roblox’s thumbnail personalization system uses multi-arm bandit instead of the more common A/B testing, Zheng said.
An A/B test sets up two alternatives for users to choose and tests which one is more popular. While traditional A/B testing picks one winner, Roblox’s multi-arm bandit-based system figures out which thumbnail among five or so choices works best for each audience segment within hours and delivers the best thumbnail to each user.
That means it doesn’t just choose which is most popular and discords the other choices. Rather, it tests to see where each thumbnail works with a certain type of user. Then it shows that thumbnail to the particular kind of user that is a match for it, Zheng said.
“It turns out, even for the same experience, different users can be attracted to the thumbnails for different reasons. Take a one example of an experience you may be familiar with, Dress to Impress,” Zheng said. “It’s doing very well on Roblox. But for some users, if they see a chic casual look in a thumbnail, they may say I want that, or I don’t want that. You may really want to see something elegant, like a dress. It’s all about what I want to see on the runway.”
To implement the testing, developers go to the Creator Hub, then go to the Thumbnails page under Creations and its subgroup Places. There, creators will see a new Home Page tab.
Once the developers have chosen five thumbnails to test, then Roblox starts its work. It will allocate traffic randomly to each of the thumbnails and watch the users respond. Then it aggregates the user behavior data where the user engages with the experience based on seeing a particular thumbnail.
If it is meaningful engagement, Roblox will see what groups of users opt for that thumbnail. Then it uses that to train an AI model, which will then distribute the thumbnails to certain users based on that user’s characteristics. If they’re more likely to like a certain thumbnail, Roblox will make sure they see it. This is the personalization that Roblox does to get as much engagement as possible.
“The really idea is this personalization comes right behind the thumbnail distribution so that the user can have the best evidence in front of them to make a call. Do I want to engage with this experience or not?,” Zheng said. “The objective is optimization. We look for qualified engagement.”
To be accurate, the thumbnail personalization needs statistical data. Zheng said, “There is indeed a minimum threshold of interested users that is required to get Thumbnail Personalization running in order to ensure accuracy, and this information is baked into onboarding for the product. Tens of thousands of experiences meet the threshold, and we seek to give an even greater number of creators insights in the future as the technology evolves.”
What the developer does
On the Home Page tab, creators can upload new thumbnails and then set up to five thumbnails as “active.” When they set two or more thumbnails to active, personalization will turn on. Then it will see which thumbnails lead to users choosing to play and play through a certain required time.
Initially, impressions will be split evenly across active thumbnails. Every hour, thumbnails with higher qualified play through rates (qPTR) for specific user groups will automatically receive more impressions from that group. After a few hours, creators will see per-thumbnail stats on the page.
The algorithm will continue to show the best thumbnail(s) to each user group while allocating some impressions to continue testing other thumbnails. Creators can add or remove thumbnails at any time, but it’s important to keep a diverse set of thumbnails active to let personalization adapt to changing user trends.
As an example, the experience in the image on this story tested five thumbnails and found that two performed significantly better than the rest for ages 18 and older, respectively. Every hour, the personalization algorithm allocates more impressions to the thumbnails with higher qualified play through rates.
Why Roblox wants better discovery
Roblox wants to improve discovery of new content on its platform so that developers can be more successful. If those devs are more successful, Roblox grows its audience and also grows its revenues and profits.
The goal with discovery is to ensure differentiated and diverse content surfaces on Roblox in a much more fair way, making it easier also for users to find the content that is right for them, Zheng said.
“We’ve been really putting a lot of the effort into the making of discovery of creative content on Roblox better in the last year or so. Our mission for discovery is really three fold. On one hand, we try to connect every user on Roblox with the best content and connections they can have. And then second is we try to connect every creations with the most relevant audience for them, while keeping our platform also very fresh, vibrant and exciting with the new patients and updates,” Zheng said.
In addition to thumbnail testing, Roblox has also made a number of investments into discovery, including improving recommendations to surface new, differentiated content on Roblox Home in a fair way, making it easier for users to find content that is right for them.
Roblox has also begun to augment its recommendations algorithm with thoughtful editorial in the form of Today’s Picks (on the homepage and marketplace). And the Our Charts page also includes helps people discover top trending, up and coming and content in different genres and categories.
This creates a vibrant ecosystem for users and creators.
“‘Today’s Picks’ is our dynamic way of helping users find new content on the platform, and it’s also a great incentive mechanism for creators to put up more new updates and new experiences out there,” Zheng said. “Because of those changes, we actually have seen a lot of diversity. For example, there are over 25 experiences now on Roblox with over one million daily active users, and that’s actually a 20% increase since last year.”
Each one of these experiences gets more people in them every day than Coachella, which gets 650,000 attendees across two weekends.
Developer reaction to personalized thumbnails
So far, about ten developers have tested the personalized thumbnails, Zheng said.
“This feature has completely changed how we approach thumbnail testing. We can now confidently test a handful of thumbnails simultaneously without fear that things will fall apart,” said ScriptOn, founder of Neura Studios, creator of short-form video creating/sharing experience Clip It, which has more than 100 million visits on Roblox.
The good thing is that Roblox can arrive at this kind of result in a matter of hours, not weeks.
This kind of multi-arm bandit is similar to the personalization that happens with any kind of advertisement for mobile games. Roblox itself doesn’t have ads, but it is investigating them. And the company will continue to improve its algorithm over time, Zheng said.
“It’s really really really really cool,” said Asimo3089, co-creator of the Roblox classic Jailbreak (which has had more than 7.1 billion visits).
“It’s a game changer. Finding winning thumbnails has gone from a risky multi-day endeavor to a much safer automated system that yields results in a few hours,” said Life_blox, creator of battle game Harbor Havoc (which has 185 million visits).
In a game like Driving Empire, the developer struck a deal with a major car brand. Then it made sense to put that car in the thumbnail image for the game. The diversity of thumbnails can vary based on the diversity of audiences for games.
And Harry Bienenstock, head of growth at Roblox developer Voldex, said in an email to GamesBeat, “Thumbnails and creatives play a massive role in discovery on Roblox. They function similarly to creative assets in mobile gaming marketing, where strong visuals can make or break user acquisition campaigns. On Roblox, however, it’s arguably even more critical because the platform’s surfacing algorithms heavily favor games with higher engagement, which starts with effective creatives.”
He added, “Finding creatives that convert efficiently can significantly boost a game’s visibility on the platform, leading to more traffic and better performance overall. In the past, we had to rely on ‘hacks’ to test creatives—either by testing directly on the game page to 100% of impressions or using makeshift solutions. This approach carried a lot of risk since a poor-performing creative could negatively impact our game’s surfacing in search and discovery.”
And Bienenstock said, “Now, with the ability to test up to five creatives simultaneously, we can mitigate that risk. Traffic is distributed dynamically, allowing the highest-converting creative to win, which means we’re optimizing for performance in real-time. It feels much more like running a campaign on Google Play or Meta Ads, where A/B testing creatives is part of the strategy. This new functionality is a game-changer for user acquisition and discovery on Roblox.”