***
I was not a born leader and hated the meetings that other department heads constantly gathered. I did not try to create a hype on the importance of my department. I just recruited guys with whom I could work and who had experience, unlike me. But the one I needed seriously, I could not find by headhunter. Such people do not seek work themselves, she finds them. I began to watch reports at conferences on the topic, read Habr. So it was hard to find. At the conferences there was not a single report with real results, everyone talked about new methods, but no one could demonstrate their application. They simply weren't there. When I tried to contact and ask questions, the speaker disappeared, only a couple answered that they really just calculated all this in excel. HabrĂŠ was no better, scraps of translations of Western articles were the best materials on the topic. Only comments on them were interesting.
The month flew by. But I did not know where to start, what to do with this big data, how to tie it to the tasks of the company. Management has already hinted that it is time to present a plan. So far, I have denied that we need to more accurately formulate the objectives of the project, what we want to get from it at the exit. They offered to get together and find out with the heads of departments, from which I realized that for a long time I could not hold out on such an argument of the lack of a plan. The staff found a girl who knew how to describe business processes. For all the guides, this was the first point in digitalization - first to algorithmize processes. I gave her a task, and he continued the search and went to meetings, where he continued to make a smart appearance.
From the comments I learned that there are contests for mashoba on Kagle. And the cool people in the mashoba are chopped up there not for money, but for who is cooler. I unsubscribed to several winners of contests related to the topic and began to wait. Some nicknames were already familiar to me from the comments on HabrĂŠ, and I was hoping that someone would respond. Two turned out to be employees of large companies bound by all sorts of agreements, so they neatly departed. But the most interesting people did not answer. He won the coolest Kaggle contests on user segmentation, referral systems, and even sales calculations based on 200 factors, including possible weather. That was what I was looking for! But he did not answer. I started looking for him by nickname on the net. There was no information. But I saw him mention the comments. So someone knew him. It was a chance. I asked in the comments who knew this, and one programmer answered me that he worked with him and might ask him for contacts for me.
He was invited by leading corporations, but he never worked in an office. And I havenât met anyone. Even his real photos on the Internet could not be found. I only knew his name and network contacts. It was somehow strange to offer to hire such a project for a company, but at a remote work site. Since it was the military, they understood only the office-barracks situation âfrom bell to bellâ. But there were no options, he needed someone who could make a cool car, since the company was already behind, in their opinion, with the introduction of big data, and it was necessary to overtake everyone to become the first. And I had to go all-in in a conversation with the leadership. But first, I had to talk to him. His name was Max.
Timlid
âI would like to invite you as a team leader and an architect to the team to create all sorts of algorithms on the mashobe.â You seem to be interested in this topic. The company is decent, it pays money.
- I do not work for companies, I work remotely on projects while they are interesting to me.
- But we are talking about a large project, you need to take up the task tightly, it is unlikely that this will be possible remotely.
- This is not a question for discussion. With those who can not work remotely, I do not work. Money can pay remotely. I am not going to waste time going to the office and come by all means by a certain time. This is stupid, but I do not do stupid things.
- Well, remote work will do. Are you ready to sign a permanent job contract remotely?
âIt all depends on what you want there.â
- Nothing special, you just have to make your own recommendation system for marketing, as well as segmentation of customers based on big data and all that.
- It is not interesting.
- And what are you interested in?
- Something more serious, more global, but it seems that this is not about you. Thanks for the suggestion.
- Wait, let me tell you everything as it is, and then you decide. I got in the ass - the company invited me to lead the implementation of machob methods in the company's work to increase efficiency, but I do not know what to offer. The company has everything - desire, still trust me, money. You can do anything, I just donât know what. Is this clear?
- It is clear, but not interesting. You have no task even. I advise you to start with this.
Max left the conversation. It was a failure. I barely found him, there is simply no such second tough guy in the machob. I had no chance of staying with the company. Another week, and they will call me to the carpet. I even asked for a couple of sick days to buy time and think what to do. Most likely, open a resume on a hunter.
Max showed up unexpectedly. He wrote on skype:
- Hi. I see you are a good guy and the company is kind of big. If you donât have ideas, then are you ready to let me realize my ideas?
- Of course! - without even thinking, I answered immediately. - And what ideas?
- There is an idea to automate the processes in the company completely, that's all. And in marketing, and in logistics, and in procurement. Even in the selection of personnel. And to make this large self-adjusting system for the desired result is profit. How do you like this task?
- This is even more of my wildest fantasies. But is it possible? I have never seen such projects in implementation. Has anyone already done this?
âI'm not interested in doing what someone else has already done.â I thought you understand that.
- Yes, of course, I wanted to say something else - is there really a development that allows you to do this?
- It doesnât matter if they are or not. There is something that will help us do this. Now there are reinforcement learning algorithms, maybe I heard already. If you think about it and bring them to mind, then this is a universal algorithm of everything. You set the goal as reinforcement, and the system itself finds a way to achieve it. And it doesnât matter what the task is if it is translated into a data set of the same format.
- What do I need to ask the management for the project besides your remote work? I canât even imagine how many people will be needed to make such an integrated system.
- Little. There will be one core, this is a neuron with memory. Fast cluster in the data center.
- And the people?
âWe need three python programmers who know the popular libraries of neurons, one data scientist, to prepare the data and keep track of it.â No, a couple, we will immediately work in all directions. And one specialist on high-performance servers.
- Such a specialist seems to be there, the company has its own data center.
- No, we need someone who can make the most high-performance cluster. You definitely donât have one. I know one, Iâll talk to him if heâs not busy. He will also need one special for databases in pairs, and weâll put him on the parsing network. We need a lot of information from the outside. Look for testers and analysts yourself how much you need. Perhaps enough to start.
- Iâll try to knock out such resources from the management, but I think there will be no problems.
âDidn't I tell you that my conditions are changing too?â
- No, what is changing?
- I want a percentage, a percentage of profit growth.
âYou baffle me.â They will not give a stranger to a guy on a distant place a percentage. I would have to coordinate your udalenka, and that is the problem.
- I offer the electronic brains of the company. Fully managing it, distributing tasks to managers and monitoring their implementation. It will be a supersystem that will decide even who to fire and who the company needs. She will have only one goal - profit. It will replace people and speed up operations, transaction costs will fall significantly. Profit will grow at a rapid pace. They cannot do this without me. Therefore, the percentage. This is true.
- I'll try. Let us briefly describe what you propose so that I can properly present your ambitions. What else should I tell them to agree to everything?
- That they will be the first.
When I tried to imagine how I would say this to the director, a stupor swept over me. I did not find a word. Unless to read written by Max on a piece of paper. I was preparing for a week, the director looked at me warily, not understanding what to expect from me. At the appointed time, I entered the meeting room, where all the directors were already sitting. The report passed in a fog. In the end, in the eyes of the meeting participants, I saw only one question - is it real or have you read fiction? The first to speak was the general:
âAnd you can realize all this?â I understand that people and time will be needed. But you understood my question.
- I cant. There is a man who can. He is the best in this business, I hardly found him. He knows his own worth and just does not agree to make such a system. Will have to go to meet him.
- Let's discuss. Well done, the report exceeded my expectations. It is hard to believe, but the goal must probably be maximum.
- If at least part of this can be realized, we will get a huge effect, I calculated here.
- Then you show, we will not delay the rest. The meeting is over.
Leaving, everyone took turns complimenting me and clapping my shoulder. Left with the general, I immediately told him about the conditions of Max in his own words. The general thought for a few seconds. âWe need to make a good contract,â he said in the end. It meant yes. He also asked to speak with each director of his part of the project and draw up a general implementation plan, preferably with timelines. He will present it to the founders. I didnât even ask about the resources, their allocation, apparently, was implied along with the approval of the project. Leaving, I was delighted with my coolness - the project was approved and together with the conditions of Max! I wrote to him right away. He answered succinctly: "I had no doubt who would refuse profit."
It was necessary to do the decomposition of the plan for months and the nearest sprints. Write applications for people. I needed statistics from analysts, documentation on processes in ERP from the development department, and much more. Everything had to be put together in order to understand where to start and what to deal with. Everyone answered friendly requests, but after a week I realized that no one was going to fulfill my requests. âThere was no time, I'll see tomorrowâ is the standard answer. And it is not clear whether this is intentionally or really all busy. In response, I myself began to receive some awkward requests. âCould you send a presentation on digitalizing our interaction with suppliers, we have a conference tomorrow.â At first I was at a loss from such requests, but in the end I began to calmly do the same as they did with my requests. Ignore. There was no documentation, the data was only in the form of reports, not raw. The only analytics program was excel. There was no talk of any uploads in BigQuery. Everything had to be done from scratch and by ourselves. The only thing that could be done quickly was to find people. And only due to the fact that I myself climbed on hh.ru and called for interviews with guys with the competencies we needed. But I did not imagine how to agree with the rest on cooperation on the project.
- Max, there are problems, I ask you to give me the data and documentation for a week, but for the time being breakfast is full. This is not a company, but some kind of swamp. Nobody needs anything, everyone is busy with their own affairs.
- Donât worry, we do not need anyone except the team you have assembled. And you need an API to raw data on customers, goods and sales, all transactions, as well as mail to customer addresses, telephony by their numbers and for now. Achieve this, go straight to the director of IT. It seems that the company needs a project only to management.
âYou, unfortunately, are right,â I answered Max with sad emoticons.
Before that, I worked only in small companies, where everyone was located in almost one room and each sought to help the other. In large corporations, this is not so. Managers of all levels try to depict vigorous activity by the number of instructions to others. But no one immediately undertakes to do what they ask. They will first ask others if they can do it. And it seemed to me that they were competing in who would come up with the most, as if they were being paid money for it. Nobody thinks about implementation anymore, the main thing is to hold a meeting and plan something. Since no one summarizes or tracks plans, 90% of such initiatives are simply forgotten in the stream of new ones. Behind this self-sufficient stream of internal information, continuously generated by managers, no one sees the client. Instead of customers reports and presentations. Kafka wrote that a large number of papers and laws are characteristic of dying empires. It was then that I got the idea that some managers have something to reduce. Now I understand why Max did not agree to go to the office.
Customer analysis
The team is assembled, and now it comes to planning sprints. On behalf of the IT directors, they provided us with some kind of documentation and made an API. With a new team, we deployed a cluster in the data center on Hadoop and began to receive data.
- Where do we start? I wrote to Max, not without optimism.
- From that simpler to work together in a team. We will do client analysis. The most understandable topic so far, and there is data. How is your website advertising organized now? How is mailing done? I donât ask about the rest, there is hardly anything else.
- Until he completely understood, he puts webmaster banners on sites on the instructions of the one who asks. Banners doing marketing. The webmaster made an admin panel in order to somehow keep track of the banners and quickly remove them if asked. Letters are sent via a cloud application, analytics with addresses are uploaded, the content manager writes the text, the advertising manager sends letters after approval by his leader, he approves the others. Somehow, as I understand it.
- What, do everything with your hands? And how many different letters are sent per month?
- Two three.
- I just donât understand how a company with such an ancient approach took a significant market share. Last century. Let's start with this. I will find a suitable Java framework for creating chains of interaction. Let's take a bourgeois cloud service as an analogue, register for now and analyze what is useful for us there. Let's start cutting into tasks.
- And what will be in the core of the system?
âMashob, of course.â I already told you that everything will be built on one core of a self-learning neuron for goals. Marketing requires customer analysis in order to quickly, directly online, cluster users by their parameters and actions on the site or in the mail. We will construct the RFM analysis to track the stages. Weâll put tracking codes in letters and on the site, we will write everything in the database for each client. And then we wind up on it all that is necessary for automatic interaction with the client - a script for building a drag & drop interaction chain with automatic selection of a communication channel with a client, depending on where it sits. Or we send a task to the assigned manager by letter, if the client is completely deaf.
- Big plan, we have to do this for six months.
âNo, I'm not an idiot to do everything myself.â Let's do it faster.
A month later, the first prototype appeared. And it was fantastic for marketing. In the system, it was possible to make hundreds of segments by hundreds of data collected by customers, to build guaranteed contact for each segment of the interaction chain. This is when the chain first tries to show the banner to the client, if it doesnât succeed, it sends a letter, if it doesnât open, it sends pushes to the application, if it didnât look there, it sends a task to the manager assigned to the client with the text what needs to be done. All customers who need action entered the network from such segments. At the same time, even the customerâs life cycle was considered as a dynamic sign, whether he is new or experienced, how often he makes purchases, whether he has already bought everything and is not going to dump. And that too was a sign for segmenting into chains. Customer actions in response to while the banner or click in the letter was also recorded in the database, and he could immediately get into the next chain. So the client could not leave the chains for months, the main thing was not to overdo it. The first welcome-chains and on abandoned baskets we built ourselves.
The only thing the marketing needed to do was build such segments and chains and write a lot of texts and draw many hundreds of banners. What, of course, they could not do right away. Max said that a little later he will make a system for automatically generating texts of letters and product banners from the database of goods. But while it was necessary to strain the marketers. I was responsible for interacting with other departments in the team, and not only led the project.
But the focus of the client analysis system was in the machob based capabilities. Max presented them to the team personally. The system analyzed the behavior and purchases of the client and could tell in advance that the client could leave. And sent the task to the manager on hold. The system knew better than managers what the client had already bought, and what was most likely to buy, based on the typical basket of such clients. This we called the âbasket approachâ. Moreover, the system itself calculated which banner or text of the letter was better to send, because it knew which text caused more response from similar ones. It was like magic for me, I first saw what a mashob can do in real business. The team started up, we worked like crazy because the results delighted us.
- There is little data about customers in your corporate system, you do not know anything about them, except for the company, position, industry and email. This is nothing. We do integration with external data providers. Request a contract with SPARK. And I will deal with the API with social networks.
- Exactly. Enrich the data. I recently saw another service that, according to comments on the social network, determines the psychotype of a person. I think this can be useful to us until I understand why, but I feel that it will not be superfluous.
- We will make recommendations to managers based on them. Give me the address. Just need to check how accurately determines. Something a little hard to believe that they can determine without special tests.
- Better than the tests, they determine, I read. Temperament is at least better determined by the reaction to people's comments, and this is full on the net. Statistically, and not some mood. And you canât fake it, as in tests.
- Well, connect, give me the address. And SPARK tighten up, for legal entities we will take info on the number in the state, turnover, founders, payments to the budget. A lot of interesting things there are also useful. Even the contacts and addresses of your managers, as it turned out, can not be trusted. They write all sorts of crap, just not to give the contacts of their customers. Very dirty data from them.
Although there was still much to be debugged, after 3 months we made a wonderful system for marketing, but for some reason no one was in a hurry to use it. I wrote letters, I gathered a meeting through the marketing director, I personally approached, but no one did segments and chains, especially letters and banners. This was the first sabotage of the system, and I did not understand why. So far, one analyst girl who works with marketers has not prompted me. We made a system too transparent. Customer analysis immediately showed how much each newsletter brought in sales, which banner they click on, and which one is useless to customers. Previously, no one could calculate the effect of a newsletter or a banner on the go, there were not even click statistics. And now everything is in full view - on the online dashboard, you can clearly see how the sales on the newsletter go. If they go. And this is the problem - no one had the practice of such online marketing, and everyone was afraid to expose their competencies. I wrote to Max.
âI said that they should all be fired,â Max answered expectably. - It's okay, you have to do it more complicated, but we can do without them.
- Any thoughts how?
- We cluster customers according to the type of activity and contacts before buying so that all customers fall into a certain segment. And we will make a universal chain that will work across all channels - in the mail, on the website or in the application. Contact accounting allows you to close chains in chains. And we will include the most important predicts - after-sales, recommendations for brands and series, outflows with discounts for returns.
- And whoever writes the texts, they do not want to make them in such quantity.
- There are a lot of texts and banners, otherwise there will be no sense. Therefore, we will make automatic product banners and texts filled with goods. Like widgets in Emarsys. Customers donât need particularly artistic texts; marketing texts are just annoying.
- So marketers will be completely without work.
- And do not forget to report this to the management that the system works by itself. Without them. As we promised. And tell marketers: âto the labor exchange, baby.â
This was Max's favorite slogan from some time when he himself believed in the viability of his algorithms. He had a goal, which was the subject of an agreement with management - to reduce costs by reducing manual operations. If we automate the creation of letters and banners, this will be the first major success of the project.
Continued in the next post ...
(c) Alexander Khomyakov alexander.xom@gmail.com