Why do we write business logic in Lua
Hi, Habr. In this post we want to talk about how and why we use a programming language with the beautiful name Lua at IPONWEB.
Lua is a script-based embedded programming language with a free interpreter and open source in C. It was developed in 1993 in Brazil, in the Tecgraf division of the Catholic University of Rio de Janeiro, and its ancestors were DEL (Data-Entry Language) and SOL (Simple Object Language) developed there earlier. One of the progenitors, the SOL language, indirectly participated in the “baptism” of the newborn - “Sol” is translated from Portuguese as “sun”, and the new language was named “Lua”, “moon”.
The ease of embedding Lua in engines written in “system” languages ​​has made it a popular scripting language for video games. For example, scripts are written in Lua in Grim Fandango and Baldur's Gate. Those who play World of Warcraft have also probably heard about Lua more than once or twice - it is on it that add-ons for the game are written that make life easier for hardcore players, casual fans, fans to measure their effectiveness and other inhabitants of the game world. Outside of gamedev, Lua is used as the scripting language of embedded systems (TVs, printers, car panels), as well as applications, for example, the VLC Media Player. Lua uses tools such as Tarantool, Redis, and OpenResty as an embedded language. Lua was also used as an extension language for Fortran computational codes simulating the thermomechanical behavior of nuclear fuel.
IPONWEB is a developer of highly loaded platforms for companies working in the field of online advertising: DSP, SSP, advertising agencies and advertisers. We spoke in detail about our work in this article . At first, we developed the business logic of our platforms in C ++, but quickly realized that this was not the best choice. To minimize costs, the platform’s performance is important, as well as the development speed, and C ++ development turned out to be too slow for us, and the complexity of adding functionality also affected. We decided to isolate the interpretation of business logic from low-level server code, began to look for a suitable language for this, and settled on Lua. It was in 2008, JavaScript implementations that were suitable for us did not exist yet, Perl, Python, and Ruby were too slow and not easy to integrate. And there was the Lua language, not very well known, but popular in game dev, and what we wanted was similar to the needs of game developers - we needed a fast engine for low-level operations and an easy-to-build fast language for business logic.
Lua is really a very fast language. An additional increase in speed can be achieved by using LuaJIT , the runtime environment for Lua 5.1, which includes a tracing JIT compiler (we use our own fork, which we already wrote about ). Since we write business logic for RTB systems, speed is critical for us: in RTB, on average, there are 120 milliseconds to process each incoming request. At the same time, only 10-15 milliseconds are allocated for code execution, and the rest of the time is waiting for a response from other services. If, for example, the user does not even notice a half-second delay when loading a site on the network, then for RTB these 500 milliseconds are a huge period of time. The answer to the question, how fast is Lua language, is this: it is fast enough so that we can write business logic on it for many years and remain in the RTB business. If the language we chose was not fast enough, we would have no one to write the platform for. Does this mean that RTB cannot be written in other languages? Does not mean. But we write RTB in Lua and successfully cope with our and client business tasks. A good example of the speed of Lua on the server is this OpenResty benchmark .
Lua as an embedded language has many advantages: it is minimalistic, compact, with a very small standard library. Its functionality is completely duplicated in C, which provides an easy and “seamless” interaction between Lua and C. Lua has a rather low login threshold compared to many other languages: most programmers coming to work in IPONWEB have never written to Lua before, but they have enough a few days to fully engage in the work.
Here is a simple example of targeting an advertising audience.
And this is how a simple handler (event handler) looks like.
The simplicity of Lua not only provides rapid development, but also allows you to do a lot of work with little effort. The IPONWEB platform is a general solution that is tailored to the needs of a particular client, while one developer and one manager can conduct the project. Read the code on Lua can not only the developers themselves, but also project managers, and sometimes customers. Together, we quickly discover the problem, find its cause and solution. Often, the project manager informs the developer about the problem and immediately suggests a way to solve it.
At the same time, the simplicity of Lua can be deceiving, while minimalism and compactness have a downside. If the code is written, for example, in Perl or Python, the developer has at his disposal huge repositories of ready-made modules, Ruby has RubyGems, and many other languages ​​have rich repositories. And Lua has LuaRocks and the three thousand modules that lie there. In addition, even if LuaRocks has the right module, it is likely that you will have to work hard to use it in a particular company. Lua provides good tools for creating a secure environment for executing code (sandboxes), and when working in sandboxes, some functions can be disabled. This means that LuaRocks modules may not work if they use functions that are blocked by the company's secure environment. This is the price of compactness and embeddability, but it’s worth the price - languages ​​with batteries, such as, for example, Python, are built in no more complicated way than Lua.
The basis of our platform is a customizable HTTP server with a convenient and extensible API that provides a set of functions for the Lua developer and is tailored for the tasks of the advertising market. This server processes hundreds of millions of requests and writes terabytes of logs per day. Incoming requests are evenly distributed across the system threads, and inside the system threads are sandboxes.
When a request arrives on the server, a coroutine is created inside the sandbox into which this request fell, which processes the request. Coroutines work independently of each other, each coroutine being created is queued for execution. The lifetime of each coroutine (the total processing time of the request, taking into account the expected response of the services involved: database, metrics, budget server) should not exceed 120 milliseconds.
In short, the request processing process can be described as follows:
The use of corutin is a separate interesting topic that deserves a detailed discussion. For example, this article details how corutins can be used to create cutscenes in video games. And the coroutines on the application server should devote a separate article, and perhaps in the future we will do it.
And then, perhaps, the use of Lua in IPONWEB will be expanded. We have ideas on how you can still use Lua in our business, and when these ideas are implemented, we will definitely share new experiences. The convenience and capabilities of Lua as an embedded scripting language can help us, in particular, speed up the processing of client data. But this is still from the field of plans and prospects.
Summing up, we can say that our choice of the language of business logic, made 11 years ago, continues to justify itself, allowing us to successfully cope with our own business tasks and help us in solving the problems of our customers. Easy to read, easy to build in, fast and easy to learn, Lua has been and remains one of the best scripting languages, the scope of which is by no means limited to game development. The IPONWEB business logic written on it is just one example of this.
Lua is a script-based embedded programming language with a free interpreter and open source in C. It was developed in 1993 in Brazil, in the Tecgraf division of the Catholic University of Rio de Janeiro, and its ancestors were DEL (Data-Entry Language) and SOL (Simple Object Language) developed there earlier. One of the progenitors, the SOL language, indirectly participated in the “baptism” of the newborn - “Sol” is translated from Portuguese as “sun”, and the new language was named “Lua”, “moon”.
The ease of embedding Lua in engines written in “system” languages ​​has made it a popular scripting language for video games. For example, scripts are written in Lua in Grim Fandango and Baldur's Gate. Those who play World of Warcraft have also probably heard about Lua more than once or twice - it is on it that add-ons for the game are written that make life easier for hardcore players, casual fans, fans to measure their effectiveness and other inhabitants of the game world. Outside of gamedev, Lua is used as the scripting language of embedded systems (TVs, printers, car panels), as well as applications, for example, the VLC Media Player. Lua uses tools such as Tarantool, Redis, and OpenResty as an embedded language. Lua was also used as an extension language for Fortran computational codes simulating the thermomechanical behavior of nuclear fuel.
Why Lua?
IPONWEB is a developer of highly loaded platforms for companies working in the field of online advertising: DSP, SSP, advertising agencies and advertisers. We spoke in detail about our work in this article . At first, we developed the business logic of our platforms in C ++, but quickly realized that this was not the best choice. To minimize costs, the platform’s performance is important, as well as the development speed, and C ++ development turned out to be too slow for us, and the complexity of adding functionality also affected. We decided to isolate the interpretation of business logic from low-level server code, began to look for a suitable language for this, and settled on Lua. It was in 2008, JavaScript implementations that were suitable for us did not exist yet, Perl, Python, and Ruby were too slow and not easy to integrate. And there was the Lua language, not very well known, but popular in game dev, and what we wanted was similar to the needs of game developers - we needed a fast engine for low-level operations and an easy-to-build fast language for business logic.
Lua is really a very fast language. An additional increase in speed can be achieved by using LuaJIT , the runtime environment for Lua 5.1, which includes a tracing JIT compiler (we use our own fork, which we already wrote about ). Since we write business logic for RTB systems, speed is critical for us: in RTB, on average, there are 120 milliseconds to process each incoming request. At the same time, only 10-15 milliseconds are allocated for code execution, and the rest of the time is waiting for a response from other services. If, for example, the user does not even notice a half-second delay when loading a site on the network, then for RTB these 500 milliseconds are a huge period of time. The answer to the question, how fast is Lua language, is this: it is fast enough so that we can write business logic on it for many years and remain in the RTB business. If the language we chose was not fast enough, we would have no one to write the platform for. Does this mean that RTB cannot be written in other languages? Does not mean. But we write RTB in Lua and successfully cope with our and client business tasks. A good example of the speed of Lua on the server is this OpenResty benchmark .
Lua as an embedded language has many advantages: it is minimalistic, compact, with a very small standard library. Its functionality is completely duplicated in C, which provides an easy and “seamless” interaction between Lua and C. Lua has a rather low login threshold compared to many other languages: most programmers coming to work in IPONWEB have never written to Lua before, but they have enough a few days to fully engage in the work.
Here is a simple example of targeting an advertising audience.
-- deal: , , -- ( , ..) -- imp: (impression opportunity), -- local function can_be_shown(deal, imp) local targeting = deal.targeting -- if not targeting then return true end -- -- (, , ): if targeting.media_type then if not passes_targeting(targeting.media_type, imp.details.media_type) then return false end end -- : if targeting.size then if not passes_targeting(targeting.size, imp.details.sizes) then return false end end return true end
And this is how a simple handler (event handler) looks like.
local adm_cache = require 'modules.adm_cache' -- adm = "ad markup" local config = require 'modules.config' local util = require 'modules.util' local AbstractHandler = require 'handlers.abstract' local ImpRenderHandler = AbstractHandler:new({is_server_request = false}) local IMP_RENDER_TEMPLATE = config.get('imp_render_template') local IMP_RENDER_BILLING = {name = 'free', type = 'in'} -- , . -- ( , ) . -- . function ImpRenderHandler:handle(params) local user_id = self.uuid local cache_id = get_param('id') if not cache_id or cache_id == '' then return self:process_bad_request({reason = '"id" parameter is expected', user_id = user_id}) end local adm = adm_cache.get(cache_id) if not adm then return self:process_bad_request({reason = 'No adm in cache', user_id = user_id}) end update_billing(IMP_RENDER_BILLING) self:log_request('imp_render', {adm = adm, user_id = user_id, cache_id = cache_id}) local content = util.expand_macro(IMP_RENDER_TEMPLATE, {ADM = adm}) return {200, content = content} end return ImpRenderHandler
The simplicity of Lua not only provides rapid development, but also allows you to do a lot of work with little effort. The IPONWEB platform is a general solution that is tailored to the needs of a particular client, while one developer and one manager can conduct the project. Read the code on Lua can not only the developers themselves, but also project managers, and sometimes customers. Together, we quickly discover the problem, find its cause and solution. Often, the project manager informs the developer about the problem and immediately suggests a way to solve it.
At the same time, the simplicity of Lua can be deceiving, while minimalism and compactness have a downside. If the code is written, for example, in Perl or Python, the developer has at his disposal huge repositories of ready-made modules, Ruby has RubyGems, and many other languages ​​have rich repositories. And Lua has LuaRocks and the three thousand modules that lie there. In addition, even if LuaRocks has the right module, it is likely that you will have to work hard to use it in a particular company. Lua provides good tools for creating a secure environment for executing code (sandboxes), and when working in sandboxes, some functions can be disabled. This means that LuaRocks modules may not work if they use functions that are blocked by the company's secure environment. This is the price of compactness and embeddability, but it’s worth the price - languages ​​with batteries, such as, for example, Python, are built in no more complicated way than Lua.
How it works?
The basis of our platform is a customizable HTTP server with a convenient and extensible API that provides a set of functions for the Lua developer and is tailored for the tasks of the advertising market. This server processes hundreds of millions of requests and writes terabytes of logs per day. Incoming requests are evenly distributed across the system threads, and inside the system threads are sandboxes.
When a request arrives on the server, a coroutine is created inside the sandbox into which this request fell, which processes the request. Coroutines work independently of each other, each coroutine being created is queued for execution. The lifetime of each coroutine (the total processing time of the request, taking into account the expected response of the services involved: database, metrics, budget server) should not exceed 120 milliseconds.
In short, the request processing process can be described as follows:
- Each received request is parsed and passes a standard check for correctness.
- Corutin is launched, which is responsible for processing this request. Inside each sandbox, there are many coroutines in different statuses.
- Request processing starts, which can have two results:
- Processing completed successfully.
- Corutin transfers control to the server. This usually happens when coroutine is waiting for a response from other services. In such cases, the work of the corutin is suspended until a response arrives or the wait time expires. When transferring control, the server starts processing the next request. This can be either a new request or a request that receives a response from all the services involved and is ready to continue executing the code. The order of processing requests is determined by the requirements of business logic.
The use of corutin is a separate interesting topic that deserves a detailed discussion. For example, this article details how corutins can be used to create cutscenes in video games. And the coroutines on the application server should devote a separate article, and perhaps in the future we will do it.
What's next?
And then, perhaps, the use of Lua in IPONWEB will be expanded. We have ideas on how you can still use Lua in our business, and when these ideas are implemented, we will definitely share new experiences. The convenience and capabilities of Lua as an embedded scripting language can help us, in particular, speed up the processing of client data. But this is still from the field of plans and prospects.
Summing up, we can say that our choice of the language of business logic, made 11 years ago, continues to justify itself, allowing us to successfully cope with our own business tasks and help us in solving the problems of our customers. Easy to read, easy to build in, fast and easy to learn, Lua has been and remains one of the best scripting languages, the scope of which is by no means limited to game development. The IPONWEB business logic written on it is just one example of this.
All Articles