Throughout my work in product development for over 8 years, I have worked in various segments: marketing, analytics, product management. Since last year I have been acting as head of a products Growth team at Parimatch Tech. We have performed more than 200 experiments together with my team, and we have tested new features as well as enhanced product and business performance. That said, why experiments are so important? You need to understand if certain product innovations will work at all before you launch them, so you would not be wasting your time and money
This article will prove very useful for owners, CEOs, and the staff of product companies who are not willing to spend months on development and much money to test their concepts. Experimentation allows you to accomplish tasks both fast and cheap. We've taken this approach as a working practice. Now, we are going to see in details how it works.
How to launch a test
The easiest, cheapest, and fastest way to test your hypotheses is through experimentation. Before launching a product, you need to understand whether it will actually work or not before moving forward and scaling it. Sometimes product companies, once they start considering the effectiveness of a new feature, rush to implement it, without even bothering to test its effectiveness and the options for handling it. So, for example, an executive discovered that a competitor or other product has a certain feature, and then decides to introduce the same feature into their own product line. However, they do not understand exactly what value it will bring to the consumer and whether it will deliver any benefit at all. The only thing that we see in this case is the desire to just do something without any feasibility analysis. You may already be aware of the challenge/demand, but might not be able to assess the effectiveness of the designed solution before the feature is fully deployed. However, you can always find a method to test the solution before you put it into production without going over budget or violating deadlines.
If you are not testing your ideas and coming up with new solutions, you cannot move forward quickly and efficiently, but there is nothing wrong with making mistakes. Through experimental practice, we discover many different product problems and find many ways to solve them. So, if something does not work during a test, it is useful to consider your "failure" in the future, and not unnecessarily waste time and money. So what hinders companies from trying experiments? The first reason is the lack of time. Second, that there is simply no such practice in the daily activities of the team.
A lot of people think that experiments go just like coming up with a great idea, and then just checking if it works. Well, that doesn't have anything to do with reality. These experiments have a clear and orderly process behind them. An underlying strategy is to identify the proper goals, develop hypotheses, test them, and then focus close to growth.
Every experiment has four basic components: hypothesis, action, data, and conclusions. That is, first we create a theory, then we test it, analyze the collected data, and then we draw conclusions.
The structure of our teamwork is organized in such a way to allow us to generate ideas collaboratively. First, we determine the direction of product growth: we hold a group session, try to see the product through the user's eyes, and test the product functionality (which may affect the parameter that we wish to improve). Next, we team up with other colleagues and share what problems we've identified and hypotheses we have for fixing them. During a brainstorm, the teams are asked to point out some missing aspects. We then prioritize all the assumptions based on the criteria of the ICE system: impact, confidence, easiness of implementation. It is the latter requirement that is essential. In some growth teams, they don't even launch an experiment when it takes longer than 5 days to complete.
A major part of the team that carries out tests are developers. These are the ones who have a good technical understanding of things. They participate both in generating ideas, hypothesis prioritizing and their implementation. We used to have only front-end developers in our team, since we only found all the experiments on the front-end and then handed them over to the main team for further implementation. Not so long we have recruited a back-end specialist, which allows us to correctly integrate into the product verified ready-made solutions.
How it all works in practice
It is common that your staff does not really understand the benefits of testing. They would say that everything is functioning well and wonder when get the "surprising" findings. Let me make an example from our work. There was an unclear password check during registration (there were no clear instructions on what characters the password should be composed of). When we tried to discuss this with the developers, they said, that there is in the corner of the page a question mark, which you can click and read the explanation, if something is not clear for you.
Nevertheless, we decided to do a test and make a more structured password verification that explains what characters you need to enter to complete the sign-up (at least one small letter, one capital letter, and one number). Our result was beyond expectations: we got +64%to conversion rate of registrations in two hours.
Another example is the experiment concerning the necessity of increasing the minimum stake for punters. This experiment aroused discontent in the company: everyone thought that such actions could lead to undesirable results. However, we took a risk and tested it on a small group of users.
It turned out to be a terrific result because almost all the users were into it. In the end, we realized that we could be wrong in our assumptions regarding users' reactions. This mindset impedes any improvement. Which means we have to dare be brave to attempt all experiments, no matter how risky, which is best done by testing how things work on a small group of customers.
By sticking to this simple principle, we stop being those people who basically sit in an office and pretend to know everything. Rather, our team honestly acknowledges that we do not really know how our customers are going to react, so every time we test a product, we learn new insights and improve it through real-life examples.
Why it is important to fail
There exists an ideology that can be summed up as "If you've mistaken, you're not competent". That is not the case with our company. If we make a few mistakes, we can easily accept failures, especially when running tests. By making a few mistakes, we are sure to find out if our ideas are valid or not, allowing us to achieve more substantial success. By doing so, we create a culture of genuine results rather than reasonable presumptions.
Often when working with new functionality that you haven't dealt with before, there can happen failures in experiments. For example, our recent experiment involved "waking-up" the inactive users. This is something our team had never done before, and nobody had ever had this sort of practice. It was naive of us to think that we could use the app to hook a passive audience right away. To solve this problem, we loaded all the "inactive" users, divided them into groups, and ran the test. But it proved to be impossible to communicate effectively.
So, we chose to focus on the users who had visited the site less than twice a week in the past month. Previously, we have unloaded them from the database, with one-half of them participating in the test. Hardly any people visited the site during this test. Therefore, we thought that the probability of attracting inactive members with the stock is very low.
Our results were shared through our internal company channel, and ultimately the conclusion was that it was more efficient to use channels of communication to bring that target audience back to the product.
Fortunately, we've never had any disappointing tests, resulting in a decline in performance. We consider as failed those trials that did not positively change anything, but they were worth trying anyway.
Such failures do not cost us much money. The cost of the error is covered by the cost of our team's operational work. Because we tend to carry out small and short-term tests, and we don't spend much money on functionality launches. Large-scale experiments are not even run. So, long as we're not confident that a new idea will actually succeed, we refuse to devote months of development and unlimited budgets to a full launch of a new feature.
So, in this sense, missteps in experiments not only save the company budget, but also bring in additional profits.
It is important not only to learn by failure, but to share your mishaps with other teams. Because sometimes you don't even question why things work the way they do. Yet when you share lessons from failed tests within your company, other colleagues begin to acknowledge their mistakes too; this may be a responsibility problem. And then coworkers, too, may question their day-to-day routines and experiment with new ideas.
Useful tips to carry out tests
There are several rules that allow you to make experimentation a regular practice bringing good results at all levels of the company:
Set clear goals. This is the most important element. The goal should not only be the growth of performance, but also the number of tests.
Stay focused. It's important to set a specific time frame for getting an accurate metric. For instance, we can work on the registration page for three months, the next three months on something else. Many employees want to test their hypotheses as quickly as possible and launch the functionality they came up with. So, you have to constantly have the team focused. on things, which are our priority right now.
Make it simple. The slogan of our team slogan is a minimum viable product squared. The term MVP is often used in the context of new product launches. But it also relates to testing, since the sooner you get user feedback, the sooner you'll make or save money.
If we have a huge hypothesis which testing may take much time, we simplify it so that we can experiment in several steps. We can, for example, test if users are attracted by new features, so we try to imitate them: we run a slight interface change to see how much the interface has improved. The users are allowed to interact with it up to a certain stage, where they will see a message: "Sorry, we're still developing". That way we can figure out and estimate how people are interacting with the new features. Provided they are curious and active, we proceed with our work by plugging in the back-end, and in case they're not, we are not going to spend more resources.
Target your metrics of improvement. For example, when people are in charge of launching specific features, they usually do their part and move on with little attention to outcomes and specific performance metrics. So, for instance, they undertake to launch features A, B, and C within several months. Once the first feature has been completed on their part, they may not even be aware that it failed to work. In fact, the reason for this is that when the goal is new functionality, all the attention is also focused on it. But if the objective is improving business indices and specific metrics, the priority is always given to the results performance. Developing a product is a complex and time-consuming endeavor that constantly revisits what needs to be focused on at a given moment. For this reason, a metrics goal helps keep the balance right.
Where do you seek out new hypotheses and why it is important to turn them into everyday practice
Using assumptions such as "I have seen a great idea there, so it might be useful for us, too" will not produce any value. It's imperative to realize where the zone of product development ends and what precisely you want to enhance. Besides analyzing product performance and competitors, there are other possible ways to look for new ideas. This is what makes the best approach for our company:
User surveys should be a regular practice. For example, the members of our team have established a rule to have meetings with our customers every Friday to get feedback. It's challenging to turn such processes into a routine, yet it's still manageable and definitely works. You can' t have a more brilliant source of insights than your customers. Our most successful experiments have all been generated thanks to feedback.
All ideas are important. The employees are the main facilitators of experiments and creators of new ideas. Get everyone to come up with ideas to improve the product, just as long as you're setting the rules. For this purpose, announce what product you are currently working with and its deadline, and admit only such ideas.</li>
Summary
Through experiments a company can test their product improvement ideas easily, fast, and at a low cost. Prior to launching a new feature, it's always good to make sure that people actually need it. This stage of the process allows for mistakes because each such mishap doesn't waste the company's money, but saves them. At the same time, it enables the team to see their actual performance.
