Bank ratings. Participation cannot be fixed
People love ratings. How many applications, games and other things have already been done in the name of a person’s desire to be on a list a couple of lines higher than anyone else. Or than a competitor, for example. People get places in the ranking in different ways, depending on motivation and moral character. Someone will try to become better and honestly step over from # 142 to # 139, and someone will decide to deposit money and joyfully take # 21 (because the top 20 brought even more).
With companies, everything is about the same. Today we’ll talk about the banks and the ratings that these banks seek to get into. In this post I will talk about common problems with research that we have in our country, about the practical difference between quantitative and qualitative tests, and how we tried to correct the current situation.
And at the end of the article is a surprise.
It all started with the fact that a year ago we started testing five banks for legal entities, choosing a pair of stylish youth banks (Modulbank and Tinkoff Bank) and three classic banks (VTB, Raiffeisenbank and Promsvyazbank). But first, a little hardware.
There are quite a few players on the market who make usability ratings for the banking sector. Namely, as many as two - Markswebb and USABILITYLAB.
And it so happened that MW and UL have now become a kind of KPI. On the one hand, this is good, since the very presence of at least something competitive sets the general motive in a rather slow market in this regard. On the other hand, it comes down mostly to functional analysis. And the motivation here from bank tops is not to make an awesome product that will take off and bring a lot of benefits to users, which will take its place in the rating, but just to be in the rating.
Your bank is in the rating = you have completed KPI = you have received a bonus. Plus, they kind of love you in the team, helped the bank get into the rating. Someone sincerely scratches Chesv. In general, there’s a lot of people, but the motivation by and large is these “bonuses” of various kinds, and not a movement towards improving the product.
And here in terms of the importance of such ratings for the market, it is important to understand one more thing. About 98% of users of banking applications do not know about these ratings at all. They frankly do not care. These ratings are for managers and executives. The remaining 2% know about ratings, but consider them a purchased topic. Once we tested the sites of banks with these dies about the first places.
People do not choose a bank for business on the basis of whether there is a die on the bank’s website with the logo of a rating or not. It’s easier for a person to call a cry from acquaintances or the FB, who uses which bank and what is satisfied / not satisfied, and limit this to social capital.
Let's start by creating a rating. To create a rating, you need to conduct a study, and here everything is usually limited to the study of one specific function, say, they test currency control.
And research costs money, and money is quite tangible. To do this qualitatively, you need to invest well - a portrait of an entrepreneur for testing costs more than the average user. Therefore, companies that try to build their revenue only on research as the main and only type of activity bear significant costs. Despite the fact that our research market is almost empty: this is not taught at universities, nor taught at schools.
By the way, about money, so that the numbers are clear. Let's say we have 20 banks in the ranking. Everyone needs to explore the top 7 functions and scenarios, spending about 1.5 hours of time. It does not make sense to conduct a test longer on one respondent, because an hour and a half is the border after which the attention is already scattered, and people get corny tired and start answering anything, just to go and have a snack and finally breathe out.
So here. It is difficult and long to take people from the base of the bank for such a study, so there remains a recruit. 5-7 scenarios for 20 banks mean that you need to recruit a minimum of 140 respondents. And then, if more than one bank will be tested on one person
The cost of one such respondent varies between 5-10 thousand rubles, there is a clear dependence on the portrait, for example, a single individual entrepreneur will cost quite inexpensively, 5 thousand. But the portrait of an exporter entrepreneur with currency control will cost about 13 thousand.
In total, there are 140 people who need to pay for participation in the study. Let us estimate the simplest and cheapest scenario, at 5,000 rubles per respondent, and we will get non-illusory 700,000 rubles. Minimum yes. Usually, this figure is close to 1,400,000. It would be time to open my own recruitment agency :)
And this is only for the main scenarios of using the bank. In addition to money, there is a more valuable resource - time. It is also spent with such a big pea on top. You can conduct tests with 30 respondents and not go crazy in 2 weeks. About 60 meetings are usually obtained per month, if you want to maintain the quality of the interview. 140 people = 2.5 person-months.
After all the respondents, it is necessary to spend another 2 months to bring the information into a digestible form - to transcribe the results, conduct analysis and grouping, make a beautiful presentation, and not the final file in Excel on a bunch of lines.
In general, it turns out about 4 months of work and 2-3 million rubles, taking into account all the costs in this period. And we still did not count taxes. And provided that no one succeeds in earning money on the research itself, such a model obviously does not look the most profitable. If you do not earn on the rating itself and places in it instead of research, by itself.
MW presentations are about 60% about functional analysis and 40% about usability. Moreover, the concept of "functional analysis" in the case of such studies is just a checklist of the presence of certain functions. You sit down, write a list of functions - so, there should be a normal payment, plus payment by photo, and also from a file, checking the counterparty, the last counterparties or payments, and so on. Then you analyze and check if there are functions from the list there or not. If there is - excellent, check the box, plus in the rating. If not, well, you get the point.
That sounds logical. But, alas, it rolls to the fact that the plus and tick with such testing is simply the presence of a function in the list, and not its quality or even the need for the user. So mobile applications began to slide in order to cram everything into itself in order to correspond to the rating, and not what the user needed. Well, that's like a dual camera at Yandex.Phone. It is there, but, they say, it does not work. But there is. In total, it turns out that 60% of the significance of such a rating is simply a tick itself, whether there is a function or not. And not how convenient and user-friendly it is.
In addition to functional analysis, there are also quantitative and qualitative studies.
Quantitative usability research will come in handy if you want to put tests on stream. You recruit more respondents, run them through the application interface, give basic tasks, and at the end just ask how it was in general and what problems were.
A high-quality usability test is much more complicated - you need to stretch the perception of the whole process and literally all the elements in the process using the Think Aloud method. All thoughts and questions that arise in people, all texts and elements incomprehensible to them. And all the root causes - why it is not clear, but how do you expect it to be named, and what word do you keep in mind?
Knowing the root causes of perception, you do not just say:
People did not find it - an unusual placement.
Do you understand how to change:
The user is looking for this element not at the bottom as we placed it, but in the upper right corner of the screen. Searches for the word “Search”, and we have “Enter”, it searches for the magnifying glass icon, and we have a button “Search”.
To summarize, after a quantitative usability test, you will have a list of problems in its most general form. Say, "User did not master finding Search." Why not mastered? But he simply did not master it - this test will not give an answer.
And after a quality test, you will have both a problem and its root cause. In the case of Search, you will have a script, the user will tell you exactly how to search for Search, what elements he expected to see and where, which words came to his mind when he did not find Search, and so on.
When you have the root cause of the problem and its detailed description, you can already fix something, change the interface so that it meets the expectations of users and solves the problems that arise.
Of course, quality ones cost more. Instead of a task and a questionnaire, you need to train a person who will conduct such tests. Take a person with the correct background, enter him into the sphere you are exploring. It takes about 3-6 months. There are few ready-made specialists in the market - that is, practically none.
But even if all these tests are carried out normally, we get the following situation - the country does not know what to do with these studies and reports. On the market, this is still referred to as some kind of ephemeral entity, they believe that they are buying just a presentation, not a solution to a problem.
Because it turns out: I ordered the bank to test, received in return some kind of superficial presentation, which is unclear how to apply or "we all knew this ourselves." What's next? But nothing, put her on the table and rejoice that she is. Because people don’t know what to do with this presentation, how to use it to improve the product, how to turn the conclusions described in it into new interfaces that will no longer be so problematic. If you do not give the depth and root causes of problems, then you do not understand how to work with problems.
In general, it’s rather sad, yes, but this does not mean that the situation cannot be corrected. Our goal was to investigate well those things in which we already had a good examination. For example, about the work of payments in the application, we had certain statistics on it. We wanted to take the main scenarios and not just print them on the "Yes - No", but to understand what kind of problems people have, at what stages, and in general - why they arise.
Distribution according to the main scenarios of legal entities
This may be a set of barriers that does not depend much on the bank itself, just giving some kind of function is made for people is not very clear.
And, of course, we wanted to do a volumetric study, and not compare a couple of banks with each other. We believed that it would then be possible to sell these detailed studies, and at the same time test the general demand for them.
Of course, the first pancake we got with a couple of lumps.
Nevertheless, we tried to take all the scenarios and go through them with one respondent. Spoiler - he survived. Perhaps now it uses banking applications much less often. But we once again confirmed the thesis that in an hour and a half everything should be minimized and another launched. Therefore, we moved from deep testing of all features to see how people find certain functions, what they pay attention to, how they perceive the structure of the main page.
Distribution of the use of platforms by individuals
When you test banking applications, you cannot just take and run them in guest mode to draw conclusions. You must at least have a bank account in order to understand how everything works there. And in the case of the bank, entrepreneurs need a live account, with a history, with a company established there. If you are also testing currency control and other joys, you will need currency accounts and a bit of afobazole. The balance cannot be empty, the transaction history should be more serious than “I’ll throw it from my account into the account 200 r, let's see how it is”.
We thought that registering accounts with all banks under study and making money on them would be a fairly quick task.
Sometimes it dragged on for a couple of weeks. From the banks, yes. And we still tested 5 banks, but would there be 20 of them?
But we were able to understand for ourselves the distribution of basic functions and the number of some individual and unpopular ones. Therefore, we went from the first pancake to the second call with a more refined methodology. A designer also appeared in the team, which brought the presentations to a new level. This is more important than it seems when you submit such information.
The result of the work were presentations on 100+ slides. When we did a study on four banks for individuals, we did not sell it. But the first study, on banks for entrepreneurs, was sold to see how interesting the market is in principle. We bought this from us 7 times (banks from the top 5 and several companies that sold development and design to banks), we did not give any kind of advertisement, except for posts on Facebook.
A great way, yes, if you only do research. We earn primarily by designing and designing.
Research for us is an opportunity to shape the market, because, as you see, it is almost absent. We were often asked, they say, guys, why did you post such a thing in free access, is it worth the money? But thanks to this, we can show the community what research can be in principle. Now, just to see a sample of such studies, you need to buy them. Well, or ask the one who bought it.
We publish them just like that. So that the market also understands what research is. So that customers who order research elsewhere can at least compare and validate the quality of what other companies sell them. In order for a common understanding to arise, research can be of high quality, and from them it is possible to get benefits and understanding what to do next. Actually, we are a little jarred, which is sad in our country in terms of research. Therefore, while we are trying to change the situation like this - having formed an understanding that you can get the best result
And besides the educational aspect, such studies and their publication is a good opportunity to generate leads. And here is a plus not only in the fact that customers come to us. Recently, one of our posts started prototyping a bank from the top 3. A few years ago we would really think - damn it, licked our topic and went to do something of our own.
And now we think - it's cool, they listen to us, and really try to make products better and closer to the user. Therefore, we will do such research further, qualitatively testing already individual semantic blocks of applications, and not just the entire product as a whole for some checklist.
Inside the team, this gives us increased expertise - not to go in the dark, but to understand how the basic scenarios and needs of people change (and they change in 1-2 years, imagine). And then, when you examine the opening of a bank account for entrepreneurs 3-4 times in 2 years, you have an ideal process, which it can be in the current technical constraints.
And the situation of the type “Wanted in the rating - paid for the rating - got into the rating” is still fed up. And the need for a new rating based on product quality has matured.
And for those who have read to the end of the article, here are two links to the study of banks for legal entities and the study of banks for individuals.
With companies, everything is about the same. Today we’ll talk about the banks and the ratings that these banks seek to get into. In this post I will talk about common problems with research that we have in our country, about the practical difference between quantitative and qualitative tests, and how we tried to correct the current situation.
And at the end of the article is a surprise.
It all started with the fact that a year ago we started testing five banks for legal entities, choosing a pair of stylish youth banks (Modulbank and Tinkoff Bank) and three classic banks (VTB, Raiffeisenbank and Promsvyazbank). But first, a little hardware.
Bank ratings in the Russian Federation
There are quite a few players on the market who make usability ratings for the banking sector. Namely, as many as two - Markswebb and USABILITYLAB.
And it so happened that MW and UL have now become a kind of KPI. On the one hand, this is good, since the very presence of at least something competitive sets the general motive in a rather slow market in this regard. On the other hand, it comes down mostly to functional analysis. And the motivation here from bank tops is not to make an awesome product that will take off and bring a lot of benefits to users, which will take its place in the rating, but just to be in the rating.
Your bank is in the rating = you have completed KPI = you have received a bonus. Plus, they kind of love you in the team, helped the bank get into the rating. Someone sincerely scratches Chesv. In general, there’s a lot of people, but the motivation by and large is these “bonuses” of various kinds, and not a movement towards improving the product.
And here in terms of the importance of such ratings for the market, it is important to understand one more thing. About 98% of users of banking applications do not know about these ratings at all. They frankly do not care. These ratings are for managers and executives. The remaining 2% know about ratings, but consider them a purchased topic. Once we tested the sites of banks with these dies about the first places.
People do not choose a bank for business on the basis of whether there is a die on the bank’s website with the logo of a rating or not. It’s easier for a person to call a cry from acquaintances or the FB, who uses which bank and what is satisfied / not satisfied, and limit this to social capital.
Let's start by creating a rating. To create a rating, you need to conduct a study, and here everything is usually limited to the study of one specific function, say, they test currency control.
And research costs money, and money is quite tangible. To do this qualitatively, you need to invest well - a portrait of an entrepreneur for testing costs more than the average user. Therefore, companies that try to build their revenue only on research as the main and only type of activity bear significant costs. Despite the fact that our research market is almost empty: this is not taught at universities, nor taught at schools.
By the way, about money, so that the numbers are clear. Let's say we have 20 banks in the ranking. Everyone needs to explore the top 7 functions and scenarios, spending about 1.5 hours of time. It does not make sense to conduct a test longer on one respondent, because an hour and a half is the border after which the attention is already scattered, and people get corny tired and start answering anything, just to go and have a snack and finally breathe out.
So here. It is difficult and long to take people from the base of the bank for such a study, so there remains a recruit. 5-7 scenarios for 20 banks mean that you need to recruit a minimum of 140 respondents. And then, if more than one bank will be tested on one person
The cost of one such respondent varies between 5-10 thousand rubles, there is a clear dependence on the portrait, for example, a single individual entrepreneur will cost quite inexpensively, 5 thousand. But the portrait of an exporter entrepreneur with currency control will cost about 13 thousand.
In total, there are 140 people who need to pay for participation in the study. Let us estimate the simplest and cheapest scenario, at 5,000 rubles per respondent, and we will get non-illusory 700,000 rubles. Minimum yes. Usually, this figure is close to 1,400,000. It would be time to open my own recruitment agency :)
And this is only for the main scenarios of using the bank. In addition to money, there is a more valuable resource - time. It is also spent with such a big pea on top. You can conduct tests with 30 respondents and not go crazy in 2 weeks. About 60 meetings are usually obtained per month, if you want to maintain the quality of the interview. 140 people = 2.5 person-months.
After all the respondents, it is necessary to spend another 2 months to bring the information into a digestible form - to transcribe the results, conduct analysis and grouping, make a beautiful presentation, and not the final file in Excel on a bunch of lines.
In general, it turns out about 4 months of work and 2-3 million rubles, taking into account all the costs in this period. And we still did not count taxes. And provided that no one succeeds in earning money on the research itself, such a model obviously does not look the most profitable. If you do not earn on the rating itself and places in it instead of research, by itself.
Quantitative and qualitative research, functional analysis
MW presentations are about 60% about functional analysis and 40% about usability. Moreover, the concept of "functional analysis" in the case of such studies is just a checklist of the presence of certain functions. You sit down, write a list of functions - so, there should be a normal payment, plus payment by photo, and also from a file, checking the counterparty, the last counterparties or payments, and so on. Then you analyze and check if there are functions from the list there or not. If there is - excellent, check the box, plus in the rating. If not, well, you get the point.
That sounds logical. But, alas, it rolls to the fact that the plus and tick with such testing is simply the presence of a function in the list, and not its quality or even the need for the user. So mobile applications began to slide in order to cram everything into itself in order to correspond to the rating, and not what the user needed. Well, that's like a dual camera at Yandex.Phone. It is there, but, they say, it does not work. But there is. In total, it turns out that 60% of the significance of such a rating is simply a tick itself, whether there is a function or not. And not how convenient and user-friendly it is.
In addition to functional analysis, there are also quantitative and qualitative studies.
Quantitative usability research will come in handy if you want to put tests on stream. You recruit more respondents, run them through the application interface, give basic tasks, and at the end just ask how it was in general and what problems were.
A high-quality usability test is much more complicated - you need to stretch the perception of the whole process and literally all the elements in the process using the Think Aloud method. All thoughts and questions that arise in people, all texts and elements incomprehensible to them. And all the root causes - why it is not clear, but how do you expect it to be named, and what word do you keep in mind?
Knowing the root causes of perception, you do not just say:
People did not find it - an unusual placement.
Do you understand how to change:
The user is looking for this element not at the bottom as we placed it, but in the upper right corner of the screen. Searches for the word “Search”, and we have “Enter”, it searches for the magnifying glass icon, and we have a button “Search”.
To summarize, after a quantitative usability test, you will have a list of problems in its most general form. Say, "User did not master finding Search." Why not mastered? But he simply did not master it - this test will not give an answer.
And after a quality test, you will have both a problem and its root cause. In the case of Search, you will have a script, the user will tell you exactly how to search for Search, what elements he expected to see and where, which words came to his mind when he did not find Search, and so on.
When you have the root cause of the problem and its detailed description, you can already fix something, change the interface so that it meets the expectations of users and solves the problems that arise.
Of course, quality ones cost more. Instead of a task and a questionnaire, you need to train a person who will conduct such tests. Take a person with the correct background, enter him into the sphere you are exploring. It takes about 3-6 months. There are few ready-made specialists in the market - that is, practically none.
But even if all these tests are carried out normally, we get the following situation - the country does not know what to do with these studies and reports. On the market, this is still referred to as some kind of ephemeral entity, they believe that they are buying just a presentation, not a solution to a problem.
Because it turns out: I ordered the bank to test, received in return some kind of superficial presentation, which is unclear how to apply or "we all knew this ourselves." What's next? But nothing, put her on the table and rejoice that she is. Because people don’t know what to do with this presentation, how to use it to improve the product, how to turn the conclusions described in it into new interfaces that will no longer be so problematic. If you do not give the depth and root causes of problems, then you do not understand how to work with problems.
Is it all really sad?
In general, it’s rather sad, yes, but this does not mean that the situation cannot be corrected. Our goal was to investigate well those things in which we already had a good examination. For example, about the work of payments in the application, we had certain statistics on it. We wanted to take the main scenarios and not just print them on the "Yes - No", but to understand what kind of problems people have, at what stages, and in general - why they arise.
Distribution according to the main scenarios of legal entities
This may be a set of barriers that does not depend much on the bank itself, just giving some kind of function is made for people is not very clear.
And, of course, we wanted to do a volumetric study, and not compare a couple of banks with each other. We believed that it would then be possible to sell these detailed studies, and at the same time test the general demand for them.
Of course, the first pancake we got with a couple of lumps.
Nevertheless, we tried to take all the scenarios and go through them with one respondent. Spoiler - he survived. Perhaps now it uses banking applications much less often. But we once again confirmed the thesis that in an hour and a half everything should be minimized and another launched. Therefore, we moved from deep testing of all features to see how people find certain functions, what they pay attention to, how they perceive the structure of the main page.
Distribution of the use of platforms by individuals
When you test banking applications, you cannot just take and run them in guest mode to draw conclusions. You must at least have a bank account in order to understand how everything works there. And in the case of the bank, entrepreneurs need a live account, with a history, with a company established there. If you are also testing currency control and other joys, you will need currency accounts and a bit of afobazole. The balance cannot be empty, the transaction history should be more serious than “I’ll throw it from my account into the account 200 r, let's see how it is”.
We thought that registering accounts with all banks under study and making money on them would be a fairly quick task.
Sometimes it dragged on for a couple of weeks. From the banks, yes. And we still tested 5 banks, but would there be 20 of them?
But we were able to understand for ourselves the distribution of basic functions and the number of some individual and unpopular ones. Therefore, we went from the first pancake to the second call with a more refined methodology. A designer also appeared in the team, which brought the presentations to a new level. This is more important than it seems when you submit such information.
The result of the work were presentations on 100+ slides. When we did a study on four banks for individuals, we did not sell it. But the first study, on banks for entrepreneurs, was sold to see how interesting the market is in principle. We bought this from us 7 times (banks from the top 5 and several companies that sold development and design to banks), we did not give any kind of advertisement, except for posts on Facebook.
- But you yourself wrote that this is a sure way to go negative!
A great way, yes, if you only do research. We earn primarily by designing and designing.
Research for us is an opportunity to shape the market, because, as you see, it is almost absent. We were often asked, they say, guys, why did you post such a thing in free access, is it worth the money? But thanks to this, we can show the community what research can be in principle. Now, just to see a sample of such studies, you need to buy them. Well, or ask the one who bought it.
We publish them just like that. So that the market also understands what research is. So that customers who order research elsewhere can at least compare and validate the quality of what other companies sell them. In order for a common understanding to arise, research can be of high quality, and from them it is possible to get benefits and understanding what to do next. Actually, we are a little jarred, which is sad in our country in terms of research. Therefore, while we are trying to change the situation like this - having formed an understanding that you can get the best result
And besides the educational aspect, such studies and their publication is a good opportunity to generate leads. And here is a plus not only in the fact that customers come to us. Recently, one of our posts started prototyping a bank from the top 3. A few years ago we would really think - damn it, licked our topic and went to do something of our own.
And now we think - it's cool, they listen to us, and really try to make products better and closer to the user. Therefore, we will do such research further, qualitatively testing already individual semantic blocks of applications, and not just the entire product as a whole for some checklist.
Inside the team, this gives us increased expertise - not to go in the dark, but to understand how the basic scenarios and needs of people change (and they change in 1-2 years, imagine). And then, when you examine the opening of a bank account for entrepreneurs 3-4 times in 2 years, you have an ideal process, which it can be in the current technical constraints.
And the situation of the type “Wanted in the rating - paid for the rating - got into the rating” is still fed up. And the need for a new rating based on product quality has matured.
And for those who have read to the end of the article, here are two links to the study of banks for legal entities and the study of banks for individuals.
All Articles