Recommendation engines battle - part I.

Recommendation systems are nowadays used almost everywhere and there is no doubt they can help users to choose the right product. From a business perspective, they increase business metrics and, in the end, earn money. In general, making relevant recommendation is a tough and complex problem and there are countless techniques and algorithms used in various domains.

Recently, due to dramatic changes in our lives, a rapid change in customer behavior has occurred and increased our online activities. This requires new approach — how to make effective recommendations and bring them to a new individual level. It means understanding every single customer and serving them to the best.

Along with the latest development in artificial intelligence and increasing volume in receiving users’ behavior data new methods and approaches arise. They enable a better understanding of users’ interests and consequently provide more relevant recommendations.

You are probably familiar with a situation when you as a consumer have been overwhelmed by hundreds or thousands of items and most of them been of no relevance to you. It can be very difficult to make the right decision do the actual purchase. Basically, those who better understand on an individual level user’s interests and offer relevant products, win user’s attention. This gives a crucial competitive advantage and differentiates the business from others. Offering more relevant products means an increase in user engagement or sales volumes, it is a strategic topic for many companies. As well-known consulting company, Boston Consulting Group states about personalization:

“Personalization will push a revenue shift of $ to the 15% of the companies that get it right. Be part of 15%”

Our goal was to introduce individualized recommendations in e-commerce domain for specific use. In order to provide the best results for customer, an experiment was set up as a challenge between our own engine Zoe.ai Deep Recommendation against the Amazon recommendation engine called Amazon Personalize. In this article , we would like to share our experience with this particular experiment.

A landscape of recommendation methods has rapidly been changing over the course of last few years. For simplification, recommendation methods can be divided into three “evolution generations”. First is the basics, either it can represent manual effort or rules in order to show relevant products based on e-tailers know-how or simple global models offering exact same set of items for all users like ‘bestsellers’ or ‘most visited’. Giving it no individualization at all, however these methods still have their time and place as they are simple and work quite well. If you want to bring recommendation to the next level though, you must look for something more advanced.

The second generation of recommendation methods brings a basic level of individual recommendation, models are no more global and can offer different set of items for every user. In brief, these methods consider what other users are doing and how you behavior is similar to theirs. You can imagine it as the banner “what other users may like”. These methods work well but have defined limitations. They do not work well with higher scales, and it could be limited to use them when having hundreds of thousands of users and items. Also, they do not consider the current behavior of user on the website. What the users are doing right now during their session on website (i.e., what products are they browsing) could be far more important for estimating their current intent than information about what they had bought a year ago (even more so for anonymous users where this information is not available).

The upcoming generation of recommendation methods is using the latest development in AI and the power of neural networks. These models can work with a large amount of data from detail collected on users’ behavior. Also, thanks to considering real-time behavior on the web page, the model can adjust offered recommendations in true real-time. In addition, it can be combined with previously mentioned methods, that is why we can call them hybrid.

The individualized recommendation has been introduced in the scenario where the user enters a category and browses list of items on a product listing page. Based on users’ behavior and interests (i.e., users’ interactions with product and web page) personalized items are displayed to them. The specific of this scenario is that only anonymous users are visiting the website, moreover, they have a small number of interactions on average but repeated visits in a short timeframe. The reason for this is the purpose of the web page, where retail products are being compared, evaluated, and subsequently are the best e-shops introduced for final purchase. Expectations, in terms of business metrics, have been set to increase traffic quality coming from product listing page (into product detail) and increase final conversion (transition to the selected e-shop).

In our experiment hybrid models are being used representing the latest generation mentioned above. We are going to compare own recommendation engine used by Zoe.ai and the engine from Amazon Personalize. A real-world dataset is used and fed into both engines. The final evaluation is done offline with the help of several performance metrics. Offline evaluation has its own limits for sure, but it is still a precise indicator for live performance. Finally, this experiment helps us to decide what engine should be chosen for real-world tests in production.

Amazon Personalize is a general recommendation engine developed by Amazon and provided in AWS public cloud as a service. This service allows data scientists with no deep knowledge of recommendation or machine learning domain to use the advanced engine in order to serve relevant recommendation. It supports basic methods as well as powerful hybrid approach using neural networks. On the other hand, you will probably discover that a level of knowledge about how recommendation works is still necessary. Results coming from the engine are, obviously, very dependent on the dataset you feed in and they will only be as good as your dataset. Moreover, in order to deliver reasonable production model, you need to experiment and train it repeatedly therefore a bit of machine learning knowledge could be helpful. From my personal experience, it could be slightly tricky, e.g., there is no possibility to stop and control training (updated: this functionality was added later after I had raised this concern to developers).

We use our own recommendation engine developed within Zoe.ai. Zoe.ai Deep Recommendations is also cloud service that can be easily connected to a final website and provides a hybrid approach using neural networks, a session-based approach, using detailed user behavior signals. In comparison with Amazon Personalize, Zoe.ai is end-to-end tool, so there is no need to worry about model training, computing and comparing metrics, constructing API, etc. The recommendation engine is just one part of product. One of the advantages is that our own engine can easily be configured, tailored and tweaked to specifics needs and particular behaviors. A default recommendation model with no additional configuration can be used as well, but such model could be outperformed by a model with specific layers configuration. Simply explained, a quick initial analysis with various algorithms is performed, then the final model configuration is decided based on performance metrics and expert judgment.

In next part we are going to show the experiment, evaluation and final results.

Thanks to Inigo Montoya