A / B Test errors exist

Recently, on the social news site Reddit, there were some interesting insights on how Zynga operates, and the one who published was one of the 520 employees fired. Two of the posts I want to explain in particular .

Post 1: In the game, I think their overall concept of pulling all player behavior data is good, but their reliance on it is also accepted by everyone. It makes development very analytical and intuitive, and it's easy to know where a game is fun. Although the difficulty is how to pull data about their behavior

Post 2: At some point, the company seems to switch from really trying out innovation to trying out the "old way of working." I'm not sure how many games have really a / b tested over the past year to find new fun and popular points. The company's expertise in data collection, data analytics, and game psychology can help us improve the gaming experience.

Zygna seems to suffer from the hidden costs of A / B testing.

The obvious cost of A / B testing is obvious: A / B testing takes time to design, implement and evaluate, which represents an opportunity cost that shows the benefits of doing other things during this time period Note: Opportunity costs are also known as alternative costs, alternative costs Opportunity costs For commercial companies, the opportunity to use the resources to produce other best substitutes is to use a certain amount of time or resources to produce a commodity Opportunity cost. Source; Baidu Encyclopedia). The apparent cost of A / B testing is often cited as a reason for not conducting a thorough test, but I believe these costs are generally over-emphasized.

If it takes about 30 minutes to design and implement an A / B test, then that organization does not have enough dedicated analytical infrastructure resources to conduct A / B testing with some internal tools. It's not a problem in itself: Products like DataEye exist to ease the burden on developers by providing "analytics services."

A true data-driven organization is in a state of permanent A / B testing and optimization. The opportunity cost of testing A / B in such an organization is negligible.

However, the hidden costs of A / B testing have a greater impact. A / B testing hidden costs are generated because a product team over-reliance on iteration, incremental improvements. This over-reliance binds the product firmly to the infrastructure and the defects of the product.

Progressive improvements are not completely unimportant: they usually grow as a percentage over a process, such as a game's landing page conversion, overall revenue, and session growth. This chart is a process that goes through a 10% cycle growth rate:

However, the optimization of this ongoing testing has left one team unable to focus on the real issues of their product. This is not the opportunity cost (ie, "Should we choose further A / B testing or repair the structural problems of the product?"), But not to identify or consider the weaknesses of the core product, as we believe that improvement is a panacea for panacea . This chart is the result of a 20% decline after experiencing a 10% cycle growth rate.

The hidden costs of A / B testing are not reflected in the misuse of resources, they reflect the idea that scientific data can replace product development. Products that failed the A / B test were like going for a gym with gunshot wounds: what to do is not done well, and ultimately in vain.

In fact, manufacturers with data problems have given up the treatment of real macro-level problems, giving way to the improvement of micro-processes. In fact, they did not solve the fundamental problems of the products, but accelerated the proliferation of the problems to some extent.

A / B testing is a tool, not a product development strategy, and should be used to give greater advantage to products that have had some success, rather than to make them think they are wrong about the product's intuition GRG Game Research Group Note: He is only suitable for some micro-process improvement, rather than a strategic development strategy can not solve the problem of the nature of the product).