Kir Shatrov: Shopify started with Rails and here they sincerely love this framework.
At the RubyRussia conference , Kir Shatrov will talk about Shopify architecture. How does one of the world's largest and heaviest Rails applications support business growth for 10 years without switching to microservices, Elixir and other popular alternatives? In a traditional interview before the conference, Kira was asked questions by Anatoly Zaitsev, developer of Evrone.
Tell me how you started your career?
Like many, I was engaged in programming at school. I made WordPress sites for $ 200 familiar. I learned about Ruby on Rails and realized that those tasks that take hours and days in PHP can be solved much faster. It seemed to me that it was worth jumping onto this train: I bought a book on Rails and began to learn step by step. Many steps did not work, and I abandoned classes. He returned a year later, tried to go through the online tutorials, and then everything worked out. Only later did I understand what the problem was: while the book was being written, translated, printed and taken to stores, a year or even a year and a half passed. Rails has changed a lot over this time. Naturally, the instructions no longer worked. And when I was able to read materials online and in English, I easily entered Rails and made the first hobby projects.
Then I met Oleg Balbekov, the founder of the RailsClub conference, now called RubyRussia. So I got to Evrone, where I worked for almost four years and thanks to classy and smart colleagues I was able to grow well. Evrone helped a lot to start: there was an opportunity to open open source, to grow. Then I worked at Evil Martians, did projects of a different scale - EBay, Groupon, Gett. Martians have an unusual open source culture and experiments, which is far from being in all teams. In between projects or directly on projects, people have the opportunity to engage in open source. This is how it turns out to develop Autoprefixer , AnyCable and not only. As a result, there is something to talk about at conferences. I spoke at RailsClub in Russia, RailsConf in the States and at a huge number of other large and not very events. And because I performed so much, I was noticed and invited to work at Shopify.
Tell me how you moved to Canada and how Shopify helped in the process.
This did not work out right away, there were difficulties with relocation: at that moment in Canada the government staged a strike and there was simply no one to apply for or redirect visas, but everything was allowed. It was 2013-2014, Shopify really lacked developers, and they began to transport engineers from around the world to themselves. This process is actively ongoing now. Today, the scale is such that, together with the Canadian government, Shopify has developed a program that allows you to get a work visa in three weeks. According to this scheme, about a hundred people move per year. At the same time, not only open source sensors and well-known speakers can get to work. We need developers who fit into the team and write good code.
It turns out that getting into Shopify is possible without a huge track record, if you do your job well and professionally?
Yes it is. It is also important to be able to talk about your work. Someone will say: “I fixed a bug and updated the Ruby version, added a new feature.” And you can tell about the same, but from the point of view of business development, solving its problems. Even during interviews with such companies, it is important what innovations you introduced at work, how you participated in the community: not only wrote code, but, for example, was a volunteer, helped organize a Ruby conference.
What exactly are you doing at Shopify right now?
The tasks are different. Of course I'm writing code. But a lot of my time is spent on organizing processes. For example, at the end of November there will be Black Friday. For everyone who works in e-commerce, this is the biggest event of the year. We begin to prepare in August: we need to agree with different teams on the release of new features, to agree with vendors and providers. And after black Friday passes, we enter the phase when we begin to do something new. Then I have to put on the architect's hat.
I decided for myself that I am comfortable doing different things and trying new things, not just writing code. But I know people who like to program most of all, and they would not want to communicate with participants of thirty teams and participate in the organization of processes. Everything is flexible in Shopify, inside the company people can find what they like to do just for them.
Shopify is a great platform. How many customers do you have?
The official figure is 800,000 active stores. This is not just registration (there are many more), these are living businesses.
What does Shopify as a platform give customers?
Our focus is small and medium business. Small businesses often have only an idea: they still do not know how they will sell. It is necessary to solve many problems: accounting of goods, organization of payments and delivery, selection of partners for all this. Our task is to save the time of entrepreneurs so that they take care of routine problems as little as possible. We take this job on ourselves.
If you live in a country where Shopify has full support, it means that you do not need to choose a payment system, because there is Shopify Payments. Just enter your tax details and everything will work. The same situation with sending goods: you print a sticker, glue it on the parcel and send. No need to buy stamps, pay for shipping, Shopify automatically integrates with mails. There are warehouse services: you can send your goods to the Shopify warehouse, and complex algorithms calculate which warehouse to store it in, how to ensure delivery to customers in one day. This allows small businesses to compete with Amazon and EBay.
For some time I worked on porting the project to your platform. This store had its inventory, goods, customer base. Everything worked quite conveniently: there is export / import, even third-party payments are connected in two clicks. You have a huge infrastructure. Do I understand correctly that most libraries (shopify api, shopify app, shopify co) are written in Ruby and Rails?
Yes it is.
Ruby is often blamed for the poor performance of large amounts of data. When you need to scale, often Ruby is missing. Why does Shopify use this particular technology stack?
The company began with a guy named Toby, who was fond of snowboards. Twelve years ago, he decided to write his own store for the sale of these snowboards. He was not interested in doing this in PHP, Java and XML. Then friend David showed him his new cool framework, which allowed you to quickly create web applications. The framework was called Ruby on Rails, and Toby built his own snowboard store on it. He liked the language and ideas of the framework, Toby was one of the first contributors to Rails. At that time, Rails did not even have a centralized git repository! People just exchanged new versions. So Tobias Lutke and David Heinemeyer Hansson began working on Rails. And soon Toby realized that it was much cooler to launch not your own snowboard store, but a whole platform for other stores.
Tobias Lyutke is our founder, he is still the CEO. He can be found in the office, all these fifteen years he has been working on Shopify. The company started with Rails, and it employs people who sincerely love this framework. They see how quickly they were able to build on Rails what they wanted. They see how quickly developers can do something, experiment and deliver to production.
I do not think that the company once seriously considered options for switching to something else. In my opinion, web applications will still rest against the database. Go somewhere, take something, reformat it, glue the template, put it in the cache and then give the result. It takes most of the time. Rails is great for rendering pages in 100-300 milliseconds. Of course, if you need to render for 8-10, you will have to choose something faster, for example, Go. The company has a department that deals with infrastructure, scaling and research of growth directions with our current technologies.
How do you solve problems with scaling and high loads?
We have a very typical stack: Rails, MySQL, Memcache, Redis. Surely you have worked with this on many projects. Somewhere in 2014, when the company was 10 years old, we realized that everything needed no longer creeps into one database. You can buy more powerful hardware for the MySQL server and grow vertically, but there is a limit to everything.
Then we decided that sharding would help grow horizontally. Like SaaS, where the data of one store does not intersect with the data of another, we can organize sharding quite simply. You never have to do a join between two different stores. With our sharding model, thousands of stores of different sizes and different loads live on the same shard. Shard includes not only the essence of the database, but also its Redis, its Memcache, and so on. Due to the complete isolation between shards, we break the entire Shopify into hundreds of small Shopify. Everyone can be hosted in a separate region, data center, provider, in a separate jurisdiction. If you have 100 shards, and something fell on one of them, it will affect only 1% of customers. This is very little when compared with the situation when, when one resource falls, everything falls on everyone.
This is horizontal scaling using sharding. And sharding not only one, the most important resource (database), but the isolation of all the components that stores use. Further interesting problems appear. For example, on some shard there are more shops where there is a lot of traffic, but on some less. On some more load, and on some less. We have to solve the problems of balancing stores inside the shard.
Are you migrating a store from one shard to another?
When you need to solve such problems - yes. We have a tuning, which I will describe in detail in the report. There will be a story about how we came to this scheme, how sharding works, how this approach can be applied not only to our business, but also to others. We ourselves had to develop a tool for the migration of stores between shards and data centers. Basically, migration is needed for rebalancing.
And then it becomes really interesting. Five years ago, we invested in an approach where a stand-alone Shopify instance can be run in isolation. Now we have clients who need to have a platform in a certain jurisdiction. This architecture allows us to build an instance that is isolated at only one point.
Yukihiro Matsumoto arrives at the conference. What would you like to ask him?
First, I’ll describe the context so that my question is more clear. As far as I know, Ruby development is a few individuals, less than ten people, among them no more than five key. Most are Japanese who have been doing this for a very long time. One of them can single-handedly implement such a major feature, such as guild or type annotations. Everything is tied to these people. And if one person sponsored by Cookpad or Heroku implements a key feature in a certain way, then that is exactly what it will be. But there is a bus factor.
In my opinion, the biggest advancements in Ruby that have taken place over the past couple of years have been initiated by large companies, as big problems cannot be solved alone. For example, Stripe hires people who develop typed programming languages and gives them a year to research. So it turns out Sorbet, which is not just a method of type-checking, but a whole paradigm, it can be scaled, its documentation is based on the experience of hundreds of people inside Stripe. And there are many such examples. Oracle sponsors Truffle, and several people are working to create the next generation virtual machine, reusing some of the virtual machines that have been developed for decades by dozens of smart people inside Oracle.
I would ask Matz how much he believes in realistically solving big Ruby problems with a small group of individual contributors. How much can such a model compete with examples when problems were solved with the help of a major sponsor.
We will discuss at the conference!
Recall that it will be held on September 28 in Moscow, all the details and registration on the site .
We are supported by:
Organizer - Evrone
General Partner - Toptal
Gold Partner - Gett
Silver Partners - Valarm , JetBrains , Bookmate, and Cashwagon
Bronze Partner - InSales
Tell me how you started your career?
Like many, I was engaged in programming at school. I made WordPress sites for $ 200 familiar. I learned about Ruby on Rails and realized that those tasks that take hours and days in PHP can be solved much faster. It seemed to me that it was worth jumping onto this train: I bought a book on Rails and began to learn step by step. Many steps did not work, and I abandoned classes. He returned a year later, tried to go through the online tutorials, and then everything worked out. Only later did I understand what the problem was: while the book was being written, translated, printed and taken to stores, a year or even a year and a half passed. Rails has changed a lot over this time. Naturally, the instructions no longer worked. And when I was able to read materials online and in English, I easily entered Rails and made the first hobby projects.
Then I met Oleg Balbekov, the founder of the RailsClub conference, now called RubyRussia. So I got to Evrone, where I worked for almost four years and thanks to classy and smart colleagues I was able to grow well. Evrone helped a lot to start: there was an opportunity to open open source, to grow. Then I worked at Evil Martians, did projects of a different scale - EBay, Groupon, Gett. Martians have an unusual open source culture and experiments, which is far from being in all teams. In between projects or directly on projects, people have the opportunity to engage in open source. This is how it turns out to develop Autoprefixer , AnyCable and not only. As a result, there is something to talk about at conferences. I spoke at RailsClub in Russia, RailsConf in the States and at a huge number of other large and not very events. And because I performed so much, I was noticed and invited to work at Shopify.
Tell me how you moved to Canada and how Shopify helped in the process.
This did not work out right away, there were difficulties with relocation: at that moment in Canada the government staged a strike and there was simply no one to apply for or redirect visas, but everything was allowed. It was 2013-2014, Shopify really lacked developers, and they began to transport engineers from around the world to themselves. This process is actively ongoing now. Today, the scale is such that, together with the Canadian government, Shopify has developed a program that allows you to get a work visa in three weeks. According to this scheme, about a hundred people move per year. At the same time, not only open source sensors and well-known speakers can get to work. We need developers who fit into the team and write good code.
It turns out that getting into Shopify is possible without a huge track record, if you do your job well and professionally?
Yes it is. It is also important to be able to talk about your work. Someone will say: “I fixed a bug and updated the Ruby version, added a new feature.” And you can tell about the same, but from the point of view of business development, solving its problems. Even during interviews with such companies, it is important what innovations you introduced at work, how you participated in the community: not only wrote code, but, for example, was a volunteer, helped organize a Ruby conference.
What exactly are you doing at Shopify right now?
The tasks are different. Of course I'm writing code. But a lot of my time is spent on organizing processes. For example, at the end of November there will be Black Friday. For everyone who works in e-commerce, this is the biggest event of the year. We begin to prepare in August: we need to agree with different teams on the release of new features, to agree with vendors and providers. And after black Friday passes, we enter the phase when we begin to do something new. Then I have to put on the architect's hat.
I decided for myself that I am comfortable doing different things and trying new things, not just writing code. But I know people who like to program most of all, and they would not want to communicate with participants of thirty teams and participate in the organization of processes. Everything is flexible in Shopify, inside the company people can find what they like to do just for them.
Shopify is a great platform. How many customers do you have?
The official figure is 800,000 active stores. This is not just registration (there are many more), these are living businesses.
What does Shopify as a platform give customers?
Our focus is small and medium business. Small businesses often have only an idea: they still do not know how they will sell. It is necessary to solve many problems: accounting of goods, organization of payments and delivery, selection of partners for all this. Our task is to save the time of entrepreneurs so that they take care of routine problems as little as possible. We take this job on ourselves.
If you live in a country where Shopify has full support, it means that you do not need to choose a payment system, because there is Shopify Payments. Just enter your tax details and everything will work. The same situation with sending goods: you print a sticker, glue it on the parcel and send. No need to buy stamps, pay for shipping, Shopify automatically integrates with mails. There are warehouse services: you can send your goods to the Shopify warehouse, and complex algorithms calculate which warehouse to store it in, how to ensure delivery to customers in one day. This allows small businesses to compete with Amazon and EBay.
For some time I worked on porting the project to your platform. This store had its inventory, goods, customer base. Everything worked quite conveniently: there is export / import, even third-party payments are connected in two clicks. You have a huge infrastructure. Do I understand correctly that most libraries (shopify api, shopify app, shopify co) are written in Ruby and Rails?
Yes it is.
Ruby is often blamed for the poor performance of large amounts of data. When you need to scale, often Ruby is missing. Why does Shopify use this particular technology stack?
The company began with a guy named Toby, who was fond of snowboards. Twelve years ago, he decided to write his own store for the sale of these snowboards. He was not interested in doing this in PHP, Java and XML. Then friend David showed him his new cool framework, which allowed you to quickly create web applications. The framework was called Ruby on Rails, and Toby built his own snowboard store on it. He liked the language and ideas of the framework, Toby was one of the first contributors to Rails. At that time, Rails did not even have a centralized git repository! People just exchanged new versions. So Tobias Lutke and David Heinemeyer Hansson began working on Rails. And soon Toby realized that it was much cooler to launch not your own snowboard store, but a whole platform for other stores.
Tobias Lyutke is our founder, he is still the CEO. He can be found in the office, all these fifteen years he has been working on Shopify. The company started with Rails, and it employs people who sincerely love this framework. They see how quickly they were able to build on Rails what they wanted. They see how quickly developers can do something, experiment and deliver to production.
I do not think that the company once seriously considered options for switching to something else. In my opinion, web applications will still rest against the database. Go somewhere, take something, reformat it, glue the template, put it in the cache and then give the result. It takes most of the time. Rails is great for rendering pages in 100-300 milliseconds. Of course, if you need to render for 8-10, you will have to choose something faster, for example, Go. The company has a department that deals with infrastructure, scaling and research of growth directions with our current technologies.
How do you solve problems with scaling and high loads?
We have a very typical stack: Rails, MySQL, Memcache, Redis. Surely you have worked with this on many projects. Somewhere in 2014, when the company was 10 years old, we realized that everything needed no longer creeps into one database. You can buy more powerful hardware for the MySQL server and grow vertically, but there is a limit to everything.
Then we decided that sharding would help grow horizontally. Like SaaS, where the data of one store does not intersect with the data of another, we can organize sharding quite simply. You never have to do a join between two different stores. With our sharding model, thousands of stores of different sizes and different loads live on the same shard. Shard includes not only the essence of the database, but also its Redis, its Memcache, and so on. Due to the complete isolation between shards, we break the entire Shopify into hundreds of small Shopify. Everyone can be hosted in a separate region, data center, provider, in a separate jurisdiction. If you have 100 shards, and something fell on one of them, it will affect only 1% of customers. This is very little when compared with the situation when, when one resource falls, everything falls on everyone.
This is horizontal scaling using sharding. And sharding not only one, the most important resource (database), but the isolation of all the components that stores use. Further interesting problems appear. For example, on some shard there are more shops where there is a lot of traffic, but on some less. On some more load, and on some less. We have to solve the problems of balancing stores inside the shard.
Are you migrating a store from one shard to another?
When you need to solve such problems - yes. We have a tuning, which I will describe in detail in the report. There will be a story about how we came to this scheme, how sharding works, how this approach can be applied not only to our business, but also to others. We ourselves had to develop a tool for the migration of stores between shards and data centers. Basically, migration is needed for rebalancing.
And then it becomes really interesting. Five years ago, we invested in an approach where a stand-alone Shopify instance can be run in isolation. Now we have clients who need to have a platform in a certain jurisdiction. This architecture allows us to build an instance that is isolated at only one point.
Yukihiro Matsumoto arrives at the conference. What would you like to ask him?
First, I’ll describe the context so that my question is more clear. As far as I know, Ruby development is a few individuals, less than ten people, among them no more than five key. Most are Japanese who have been doing this for a very long time. One of them can single-handedly implement such a major feature, such as guild or type annotations. Everything is tied to these people. And if one person sponsored by Cookpad or Heroku implements a key feature in a certain way, then that is exactly what it will be. But there is a bus factor.
In my opinion, the biggest advancements in Ruby that have taken place over the past couple of years have been initiated by large companies, as big problems cannot be solved alone. For example, Stripe hires people who develop typed programming languages and gives them a year to research. So it turns out Sorbet, which is not just a method of type-checking, but a whole paradigm, it can be scaled, its documentation is based on the experience of hundreds of people inside Stripe. And there are many such examples. Oracle sponsors Truffle, and several people are working to create the next generation virtual machine, reusing some of the virtual machines that have been developed for decades by dozens of smart people inside Oracle.
I would ask Matz how much he believes in realistically solving big Ruby problems with a small group of individual contributors. How much can such a model compete with examples when problems were solved with the help of a major sponsor.
We will discuss at the conference!
Recall that it will be held on September 28 in Moscow, all the details and registration on the site .
We are supported by:
Organizer - Evrone
General Partner - Toptal
Gold Partner - Gett
Silver Partners - Valarm , JetBrains , Bookmate, and Cashwagon
Bronze Partner - InSales
All Articles