PHP libraries for e-commerce: working with ATOL and Payture, parsing GS1 codes and other tasks
Hi, my name is Pavel Savelyev, I’m the head of the business process automation department at Lamoda. We work with very different tasks, and try to choose the most convenient tools for each. Accordingly, we use different languages - in our systems you can find Java, Go, and a little Kotlin for android. At the same time, a significant part of the development is conducted in PHP; more than two dozen services are written on it that automate not only work with orders, but also the operational processes of a wide delivery network, call centers in three countries and our own photo studio, as well as providing all this in the form services to our B2B partners. These services are supported and developed by 5 development teams of our department.
As the services themselves and the infrastructure around them develop, similar tasks more often arise in these systems, such as logging into the common CLS (Centralized Logging System), testing file storage, collecting metrics for Prometheus and others. We try to standardize ways to solve such problems and use common components for different systems.
So this is the process that happens on a regular basis with us, we created a guideline that allows us to conduct it most painlessly - I will try to report on this at one of the upcoming conferences.
More than two dozen of our PHP libraries have been made publicly available on GitHub. And we plan to spread further. What for? Well, we invested a lot of resources and (I want to believe) did well. And we cunningly hope that other developers will use our libraries, help them finish and develop further, instead of spending time writing their analogs from scratch. In this article I want to briefly talk about seven libraries designed to solve common tasks for e-commerce - I will be glad if they are useful to you, and I will be even more happy to jointly develop them :)
Like other Russian companies, we are obliged to fully comply with the requirements of FZ-54, one of which is online fiscalization. All orders prepaid on lamoda.ru are always fiscalized: they generate online checks that are sent to customers. Our fiscalization service works by API with the ATOL Online system , and the first library on our list is a full-fledged client for this service. In addition to the library itself, we also posted a bundle , with which you can easily connect it to any projects based on the Symfony framework. The library itself can be embedded in any other PHP framework: Laravel, Yii, etc. - just write a “wrapper” for the library.
To process prepaid payments, we actively interact with the Payture service. This service has several software interfaces. We use the Payture InPay option and have written our own API client for it. The library allows you to manipulate multiple terminals, supports standard PSR-3 logging. It is also possible to use the pre-configured Guzzle client - this makes it easy to organize testing using the Guzzle Mock Handler .
Our library bundle provides a semantic configuration of terminals and allows you to conveniently configure client settings (so far only timeouts) for various API operations.
One of the most important projects of 2019 in our company is support for state labeling of goods. As part of this project, special unique codes will be applied to all products of certain categories - in the GS1 Datamatrix format. These codes will allow any buyer to verify the authenticity of the goods, their origin and history. In order for Lamoda internal systems to work with these barcodes, we have developed a library for the correct parsing of GS1 codes.
In the near future, we also plan to lay out the source codes of our developed clients for interaction with the Marking and Traceability Information System (IP MP).
We have more than a hundred microservices that perform many separate operations: they check the status of payments or new files in the vaults, send control commands to the cashiers, download and process photos from external services. Almost all of these operations are performed in the background, and the command bus pattern is excellent for managing them. To implement the bus in PHP-systems, we chose a turnkey solution - the open library Tactician .
However, a problem arose: our background teams often interact with external services, which have a limit on the number of operations in n seconds. And Tactician does not have the ability to control the number of executable commands in a specific time window. Therefore, we developed an additional middleware - Tactician rate limit library. With its help, you can add a new processing layer that tracks the number of commands executed on the bus according to the selected rate limiting strategy. Strategies are pluggable, strategies from the stiphle library are available out of the box .
Also in the public domain is our Symfony bundle to the library.
Our microservices generate technical and business metrics, which are then collected through the Prometheus Operator from all over the k8s cluster. To manage all this, we wrote a library that processes custom metrics according to the “collect-save-show” scenario. At the same time, the library supports operating modes in which one of the script items can be omitted to increase efficiency. For example, for quick computable metrics, a simplified “collect-show” script may be executed. And work with slow business metrics can partially translate into the background, while breaking down into two stages: “collect-save” + “collect (from storage) - show”.
The library has the necessary levels of abstraction both for writing its metric generators and for writing repositories. Out of the box there is an abstract adapter for Doctrine, which can be configured on entity to save data to the database.
As metric render formats, prometheus and telegraf httpjson are currently supported.
The library comes with a Symfony bundle, which provides a semantic configuration of metric sources, repositories and routing metrics. It also has helper commands for debugging and saving metrics from sources (for example, for calculating cron metrics).
To automate testing, we use the Codeception framework, which allows us to write tests of various levels and has a fairly extensive library of standard modules. We wrote more about our approaches to test development recently in a separate article and spoke at the PHP Russia conference. Codeception has ready-made modules for interacting with FTP and the local FileSystem, but in our tests there is a need to work with more file systems. At a minimum, we are also using AWS S3 and Webdav. In addition, I would like to interact with all file systems using the same API (this is all file systems :)).
Fortunately, there is an open FlySystem library that provides a single software interface for working with different file systems. So we only needed to combine the two tools - which we did by writing a wrapper over FlySystem in the form of a Codeception-flysystem module . Now it supports SFTP, S3 and Webdav. It is enough to configure the settings once to connect to the desired file system in the yml test config, and after that you can work with all file systems using the same set of methods: write the file, copy the file, clean the directory, etc. The module is already included in the page of additions and recommendations from Codeception: codeception.com/addons .
In the department of automation of business processes, there are systems that operate in multi tenant mode. To ensure their work, it is necessary to be able to work with environment variables - to determine which variable to use at the current time.
Our library provides several strategies for working with environment variables in multi tenant mode. Based on the parameters passed at the initialization stage, the library determines which environment variable should be accessed in the current request.
This is only the first part of the libraries. We have a dozen more inside - they are waiting in line when we “comb” them a bit and put them in the public domain. It motivates me to understand that these libraries may be useful to someone else. I rejoice at the comments and stars on the github, and I hope to continue to develop libraries with other developers. Indeed, many Russian e-commerce projects work with ATOL and Payture. For Datamatrix, in addition to the code parser described in the article, we also have a couple of clients that we already use internally - these libraries are the first in the queue on GitHub.
We try not to forget about the other languages - we have already posted the first library on Go (we wrote more about it here on Habré ) and we are preparing others. Stay tuned!
As the services themselves and the infrastructure around them develop, similar tasks more often arise in these systems, such as logging into the common CLS (Centralized Logging System), testing file storage, collecting metrics for Prometheus and others. We try to standardize ways to solve such problems and use common components for different systems.
When one of the teams is faced with the adaptation or integration of a new service / tool that may become common to all, we initiate the development of the library in this team. And then the finished component is prepared for future reuse and laid out in the public domain.
So this is the process that happens on a regular basis with us, we created a guideline that allows us to conduct it most painlessly - I will try to report on this at one of the upcoming conferences.
More than two dozen of our PHP libraries have been made publicly available on GitHub. And we plan to spread further. What for? Well, we invested a lot of resources and (I want to believe) did well. And we cunningly hope that other developers will use our libraries, help them finish and develop further, instead of spending time writing their analogs from scratch. In this article I want to briefly talk about seven libraries designed to solve common tasks for e-commerce - I will be glad if they are useful to you, and I will be even more happy to jointly develop them :)
1. Online fiscalization: a client for ATOL Online
Like other Russian companies, we are obliged to fully comply with the requirements of FZ-54, one of which is online fiscalization. All orders prepaid on lamoda.ru are always fiscalized: they generate online checks that are sent to customers. Our fiscalization service works by API with the ATOL Online system , and the first library on our list is a full-fledged client for this service. In addition to the library itself, we also posted a bundle , with which you can easily connect it to any projects based on the Symfony framework. The library itself can be embedded in any other PHP framework: Laravel, Yii, etc. - just write a “wrapper” for the library.
2. Prepaid payments: interaction with Payture
To process prepaid payments, we actively interact with the Payture service. This service has several software interfaces. We use the Payture InPay option and have written our own API client for it. The library allows you to manipulate multiple terminals, supports standard PSR-3 logging. It is also possible to use the pre-configured Guzzle client - this makes it easy to organize testing using the Guzzle Mock Handler .
Our library bundle provides a semantic configuration of terminals and allows you to conveniently configure client settings (so far only timeouts) for various API operations.
3. Product labeling: GS1 Datamatrix code parser
One of the most important projects of 2019 in our company is support for state labeling of goods. As part of this project, special unique codes will be applied to all products of certain categories - in the GS1 Datamatrix format. These codes will allow any buyer to verify the authenticity of the goods, their origin and history. In order for Lamoda internal systems to work with these barcodes, we have developed a library for the correct parsing of GS1 codes.
In the near future, we also plan to lay out the source codes of our developed clients for interaction with the Marking and Traceability Information System (IP MP).
4. Microservice management: middleware for the Tactician team bus
We have more than a hundred microservices that perform many separate operations: they check the status of payments or new files in the vaults, send control commands to the cashiers, download and process photos from external services. Almost all of these operations are performed in the background, and the command bus pattern is excellent for managing them. To implement the bus in PHP-systems, we chose a turnkey solution - the open library Tactician .
However, a problem arose: our background teams often interact with external services, which have a limit on the number of operations in n seconds. And Tactician does not have the ability to control the number of executable commands in a specific time window. Therefore, we developed an additional middleware - Tactician rate limit library. With its help, you can add a new processing layer that tracks the number of commands executed on the bus according to the selected rate limiting strategy. Strategies are pluggable, strategies from the stiphle library are available out of the box .
Also in the public domain is our Symfony bundle to the library.
5. Collecting and rendering metrics for Prometheus
Our microservices generate technical and business metrics, which are then collected through the Prometheus Operator from all over the k8s cluster. To manage all this, we wrote a library that processes custom metrics according to the “collect-save-show” scenario. At the same time, the library supports operating modes in which one of the script items can be omitted to increase efficiency. For example, for quick computable metrics, a simplified “collect-show” script may be executed. And work with slow business metrics can partially translate into the background, while breaking down into two stages: “collect-save” + “collect (from storage) - show”.
The library has the necessary levels of abstraction both for writing its metric generators and for writing repositories. Out of the box there is an abstract adapter for Doctrine, which can be configured on entity to save data to the database.
As metric render formats, prometheus and telegraf httpjson are currently supported.
The library comes with a Symfony bundle, which provides a semantic configuration of metric sources, repositories and routing metrics. It also has helper commands for debugging and saving metrics from sources (for example, for calculating cron metrics).
6. Testing file storage: working with different file systems
To automate testing, we use the Codeception framework, which allows us to write tests of various levels and has a fairly extensive library of standard modules. We wrote more about our approaches to test development recently in a separate article and spoke at the PHP Russia conference. Codeception has ready-made modules for interacting with FTP and the local FileSystem, but in our tests there is a need to work with more file systems. At a minimum, we are also using AWS S3 and Webdav. In addition, I would like to interact with all file systems using the same API (this is all file systems :)).
Fortunately, there is an open FlySystem library that provides a single software interface for working with different file systems. So we only needed to combine the two tools - which we did by writing a wrapper over FlySystem in the form of a Codeception-flysystem module . Now it supports SFTP, S3 and Webdav. It is enough to configure the settings once to connect to the desired file system in the yml test config, and after that you can work with all file systems using the same set of methods: write the file, copy the file, clean the directory, etc. The module is already included in the page of additions and recommendations from Codeception: codeception.com/addons .
7. Working with environment variables in multi tenant mode
In the department of automation of business processes, there are systems that operate in multi tenant mode. To ensure their work, it is necessary to be able to work with environment variables - to determine which variable to use at the current time.
Our library provides several strategies for working with environment variables in multi tenant mode. Based on the parameters passed at the initialization stage, the library determines which environment variable should be accessed in the current request.
To be continued
This is only the first part of the libraries. We have a dozen more inside - they are waiting in line when we “comb” them a bit and put them in the public domain. It motivates me to understand that these libraries may be useful to someone else. I rejoice at the comments and stars on the github, and I hope to continue to develop libraries with other developers. Indeed, many Russian e-commerce projects work with ATOL and Payture. For Datamatrix, in addition to the code parser described in the article, we also have a couple of clients that we already use internally - these libraries are the first in the queue on GitHub.
We try not to forget about the other languages - we have already posted the first library on Go (we wrote more about it here on Habré ) and we are preparing others. Stay tuned!
All Articles