For a year and a half, I have been holding the position of the main ML developer in my company. Six months I manage a small team. I have gained experience that I want to share. I will do this in the format of a top of errors and potential difficulties.
We want a neural network!
Many people have heard at least something about artificial intelligence and the achievements of neural networks. Some people want to use neural networks in business just because it is popular. So you have solved some small task of text analysis and you can safely declare to everyone that your business uses the most modern technologies of artificial intelligence.
Moreover, almost no one understands the strengths and weaknesses of neural networks. Indeed, due to the high resource consumption, they are far from always the optimal model.
Quite often, algorithms like boosting, k-nearest neighbors or svm can show higher quality indicators in solving business problems (if we are talking about the classic script features -> value) than neural networks. This is despite the fact that they are much more lightweight.
If a client tells you: “We need a neural network to solve such and such a problem,” then most likely he simply does not know what tools are best to solve this problem. But it seems to him that some complicated algorithms are needed here and the only thing that comes to his mind is a neural network.
So you can solve the problem through some more light model.
It sounds obvious, but in fact, after such a request from the client, you can spend time sorting through various neural network architectures before you realize that the task could be solved much easier.
Morning data, evening estimeyta
Quite often, you will be asked to give deadlines for performing experiments / rolling out solutions in the prod. Moreover, as the basis for such estimates can provide only an abstract description of the problem. For example: “We want a neural network that will, according to medical images, diagnose a person with a disease. How long will it take? ” People who did not work with ML (and sometimes worked) have little understanding of the concept of experiments. Many approach the development of ML solutions as standard software development. But these are all mistakes. ML development is closer to science than to development (especially in the initial stages). Most of the time is required for data analysis and experiments. And people rarely understand this.
If you are lucky and you have, for example, an experienced project manager, then you do not have to deal with such problems. He himself will explain everything to the customer. But it happens the other way around - when the manager himself poorly understands the specifics of ML, bends under the client and starts kicking you along with him, demanding the deadlines a couple of days or a week after receiving a request from the client. Sometimes even before receiving data. Then, you most likely have to name some dates (well, or change your place of work, which is also a good option in this situation). But be careful if you still decide to make an estimate of the timeline without enough information. Take time to experiment with margin.
The accuracy of the model in the preliminary experiment is 99.99999%
Even if you obtained high metrics during preliminary experiments or when testing a prototype, this is not a reason to immediately communicate them to the client. But everything you say can be used against you.
If you received high values of model quality estimates in preliminary experiments, you can tell the client that the problem is uniquely solvable, but you should not delight him with numbers in advance that may not be so good on new pieces of data. Or you can call the metrics, but with the obligatory reservation that you can not guarantee that in other conditions the quality will remain the same. It can both improve and worsen. There is always the option that you were corny wrong when performing experiments (incorrectly written generator for example, and some of the data leaked from training to test).
Alternatively, you can fix different payments in the contract for different final quality of your model.
Generally speaking, people usually trust concrete examples more than abstract numbers. A small demonstration of your model will give the client more confidence than statements about 99% accuracy.
If you are solving a problem, for example, of detection / classification using convolutional neural techniques and it is time to write a report on the success of the team for a conditional period, spend the extra 30 minutes and make a couple of beautiful pictures: play with the visualization of convolutional filters at high levels of abstraction or hidden fully connected layers, make heat maps of classes, etc.
There will be no problems with the servers
At some point I began to notice that the work of the model often rests on its size. For example, when you need to make a classifier for 5 million classes, even the simplest models can be too resource-intensive. Then the question naturally arises of the server specification, which the client can dismiss with something like: "There will be no problems with the servers - we will choose something on the cloud services."
The problem is that it may not represent the scale of the model at all. For example, the dataset will weigh 2 terabytes, and the weight matrix of the trained model is another 500 gigabytes. To use such a model, you need 68-128 GB of RAM. To rent such a car can cost a client from several thousand to several tens of thousands of dollars per month (if more GPUs are needed). And here not many agree to pay for such servers.