Run php scripts via php-fpm without a web server. Or your FastCgi client (under the hood)

I welcome all readers of "Habr".

Disclaimer

The article turned out to be quite long and for those who do not want to read the background, but want to go straight to the point, please go straight to the chapter "Solution"

Introduction

In this article I would like to talk about solving a rather non-standard problem that I had to face during the work process. Namely, we needed to run a bunch of php scripts in a loop. I will not discuss the reasons and controversy of such an architectural solution in this article, because in fact, it wasn’t about it at all, it was just a task, it had to be solved and the solution seemed interesting enough to me to share with you, all the more I didn’t find any mana on this issue on the Internet (well, of course, except for the official specifications). Speck of course this is good and of course everything is in them, but I think you will agree that if you are not particularly familiar with the topic, and even limited in time, then understanding them is still a pleasure.

Who is this article for?

For everyone who works with the web and the FastCgi protocol only knows that this is the protocol according to which the web server runs php scripts, but wants to study it in more detail and look under the hood.

Justification (why this article)

In general, as I wrote above when we were faced with the need to run many php scripts without the participation of a web server (roughly speaking from another php script), the first thing that came to mind is ...

shell_exec('php \path\to\script.php')
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

But at the start of each script, an environment will be created, a separate process will be launched, in general, it seemed to us somehow costly in terms of resources. This implementation was rejected. The second thing that came to mind is of course php-fpm , it’s so cool, it only starts the environment once, monitors the memory, logs everything there correctly, starts and stops scripts, in general everything does cool, and of course we liked this way more.

But it’s bad luck, in theory we knew how it works, in general terms (as it turned out to be very general), but it turned out to be quite difficult to implement this protocol in practice without the participation of a web server. Reading the specifications and a couple of hours of unsuccessful attempts showed that it would take time to implement, which we did not have at that time. There’s no mana for the implementation of this venture in which this interaction could be simply and clearly described, we couldn’t take any specs either, from the ready-made solutions we found a Python script and a Pykhov lib on the github, which they did not want to drag to my project in the end (maybe it’s not correct, but not really, we love all sorts of third-party libraries and even not very popular ones, and therefore not tested). In general, as a result of this idea, we refused and implemented all this through the good old rabbitmq.

Although the problem was finally solved, I still decided to understand FastCgi in detail, and in addition I decided to write an article about it, which will simply and in detail describe how to get php-fpm to run a php script without a web server, or rather, as a web the server will have a different script, then I will call it the Fcgi client. In general, I hope that this article will help those who are faced with the same task as we do and after reading it will be able to quickly write everything as he needs.

Creative search (false path)

So the problem is indicated, we must proceed to the solution. Naturally, like any "normal" programmer, to solve a problem about which it is not written anywhere what to do and what to enter into the console, I did not read and translate the specification, but immediately came up with my own "brilliant" solution. Its essence is as follows, I know that nginx (we use nginx and so as not to write foolish things further - a web server, I will write nginx, as it is more sympathetic) transfers something to php-fpm , it something processes php-fpm based on the script starts it, well, everything seems to be simple, I’ll take it and secure it with nginx and transfer the same thing.

Great netcat will help here (UNIX-utility for working with network traffic, which in my opinion can do almost anything). So, we set netcat to listen on the local port, and configure nginx to work with php files through the socket (naturally, the socket on the same port that netcat listens on)

listening to 9000 port

 nc -l 9000
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

You can verify that everything is OK, by contacting the address 127.0.0.1:9000 through the browser, the following picture should be

configure nginx so that it processes php scripts through a socket on port 9000 (in the settings '/ etc / nginx / sites-available / default', of course, they may differ)

 location ~ \.php$ { include snippets/fastcgi-php.conf; fastcgi_pass 127.0.0.1:9000; }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

After these manipulations, we will check what happened by contacting the php script through the browser

It can be seen that nginx sent environment variables, as well as non-printable characters, that is, the data was transferred in binary encodings, which means that they cannot be so easily copied and sent to the php-fpm socket . If you save them to a file, for example, then they are saved in hexadecimal encoding, it will look like this applies

But this also does not give us much, probably purely theoretically they can be converted to binary encoding, somehow (I can’t even imagine how) to send them to the fpm socket, and there is even a chance that this whole bike will somehow work, and even start some kind of a script, but somehow it’s all ugly and awkward.

It became clear that this path was completely wrong, you can see for yourself how miserable it all looks, and even more so, all these actions will not allow us to control the connection, nor will they bring us closer to understanding the interaction between php-fpm and nginx .

Everything is gone, the study of the specifications can not be avoided!

Solution (here all the salt of this article actually begins)

Theoretical training

Let’s now look at how all the same there is a connection and data exchange between nginx and php-fpm . A little theory, all communication takes place as is already clear through sockets, we will further consider specifically a connection through a TCP socket.

The unit of information in the FastCgi protocol is a cgi record . The server sends such records to the application and receives exactly the same records in response.

A bit of theory (structure)

Next, consider the structure of the record. To understand what a record consists of, you need to understand what C structures are like and understand their designations. For those who do not know further, this will be briefly (but enough for understanding) described. I’ll try to describe as simply as possible, it’s pointless to go into details, and I’m afraid that I will get confused in the details, the main thing is to have a common understanding.

Structures are simply a collection of bytes, and a notation for them allows them to be interpreted. That is, you just have a sequence of zeros and ones, and some data is encrypted in this sequence, but so far you have no annotation for this sequence, then this data does not represent any value to you, because you cannot interpret them.

 //     1101111000000010010110000010011100010000
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

What is visible here, we have some bits, what kind of bits we have no idea. Well, let's try for example to divide them into bytes and represent in decimal system

 //   5  11011110 00000010 01011000 00100111 00010000 //    222 2 88 39 16
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Well, we interpreted them and got some results, let's say that these data are responsible for how much a certain apartment owes for electricity. It turns out that in house 222 apartment number 2 must pay 88 rubles. And what else for two digits, what to do with them just to drop? Of course not! the fact is that we did not have a notation (format) that would tell us how to interpret the data, and interpret it in our own way, in this regard we received not only useless, but also harmful results. As a result, apartment 2 paid absolutely not what it should have. (the examples are certainly far-fetched and serve only to more clearly explain the situation)

Now let's see how we should interpret this data correctly, having a notation (format). Further I will call a spade a spade, namely notation = format ( here formats ).

 //  "Cnn" //  //C -   (char) (8 ) //n -  short (16 ) //      11011110 0000001001011000 0010011100010000 //    222 600 10000
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Now everything converges in the house No. 222 apartment 600 for electricity should be 1000 rubles. I think now the importance of the format is clear, and now it’s clear how roughly a similar structure looks like. (please pay attention, here the goal is not to explain in detail what these structures are, but to give a general understanding of what it is and how it works)

The symbol of this structure will be

 struct { unsigned char houseNumber; unsigned char flatNumperA1; unsigned char flatNumperA2; unsigned char summB1; unsigned char summB2; }; // ,           // houseNumber -  // flatNumperA1 && flatNumperA2 -  // summB1 && summB2 -  
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

A little more theory (FastCgi entries)

As I said above, the unit of information in the FastCgi protocol is records. The server sends the records to the application and receives the same records in response. A record consists of a header and a body with data.

Header structure:

1. protocol version (always 1) is denoted by 1 byte ('C')
2. record type. To open, close the connection, etc. I will not consider everything, then I will only consider what is needed for a specific task, if others are needed, welcome the specification here. It is indicated by 1 byte ('C').
3. Request ID, an arbitrary number, indicated by 2 bytes ('n')
4. the length of the body of the record (data), indicated by 2 bytes ('n')
5. the length of the alignment data and the reserved data, one byte each (here you do not need to pay special attention so as not to be distracted from the main thing in our case there will always be 0)

Next is the body of the record:

1. the data itself (here it is precisely the variables that are transferred) can be quite large (up to 65535 bytes)

Here is an example of the simplest FastCgi binary recording with format

 struct { // unsigned char version; unsigned char type; unsigned char idA1; unsigned char idA2; unsigned char bodyLengthB1; unsigned char bodyLengthB2; unsigned char paddingLength; unsigned char reserved; //  unsigned char contentData; // 65535  unsigned char paddingData; };
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Practice

Script client and transmitting socket

For data transfer we will use the standard php socket extension. And the first thing that needs to be done is to configure php-fpm to listen on the port on the local host, for example 9000. This is done in most cases in the file '/etc/php/7.3/fpm/pool.d/www.conf', the path of course Depends on your system settings. There you need to register something like the following (I bring the whole footcloth so that you can navigate, the main section is listen here)

 ; The address on which to accept FastCGI requests. ; Valid syntaxes are: ; 'ip.add.re.ss:port' - to listen on a TCP socket to a specific IPv4 address on ; a specific port; ; '[ip:6:addr:ess]:port' - to listen on a TCP socket to a specific IPv6 address on ; a specific port; ; 'port' - to listen on a TCP socket to all addresses ; (IPv6 and IPv4-mapped) on a specific port; ; '/path/to/unix/socket' - to listen on a unix socket. ; Note: This value is mandatory. ;listen = /run/php/php7.3-fpm.sock listen = 127.0.0.1:9002
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

After setting up fpm, the next step is to connect to the socket

 $service_port = 9000; $address = '127.0.0.1'; $socket = socket_create(AF_INET, SOCK_STREAM, SOL_TCP); $result = socket_connect($socket, $address, $service_port);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Start of FCGI_BEGIN_REQUEST request

To open a connection, we must send a record with type FCGI_BEGIN_REQUEST = 1 The title of the record will be like this (to convert the numeric values ​​to a binary string with the specified format, the php function pack () will be used)

 socket_write($socket, pack('CCnnCx', 1, 1, 1, 8, 0)); //  - 1 //  - 1 - FCGI_BEGIN_REQUEST //id - 1 //   - 8  // - 0
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

The recording body for opening a connection must contain a recording role and a flag controlling the connection

 //      //struct { // unsigned char roleB1; // unsigned char roleB0; // unsigned char flags; // unsigned char reserved[5]; //}; //php  socket_write($socket, pack('nCxxxxx', 1, 0)); // - 1 -  // - 1 -    1   
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

So, the record for opening the connection was sent successfully, php-fpm will accept it and will continue to expect from us a further record in which we need to transfer data to deploy the environment and run the script.

Passing Environment Parameters FCGI_PARAMS

In this record, we will pass all the parameters that are needed to deploy the environment, as well as the name of the script that we will need to run.

Minimum required environment settings

 $url = '/path/to/script.php' $env = [ 'REQUEST_METHOD' => 'GET', 'SCRIPT_FILENAME' => $url, ];
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

The first thing we need to do here is to prepare the necessary variables, that is, the name => value pairs that we will pass to the application.

The structure of pairs name value will be such

 //          128  typedef struct { unsigned char nameLength; unsigned char valueLength; unsigned char nameData unsigned char valueData; }; //    1 
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

There is 1 byte first - the name is long, then 1 byte is the value

 //         128  typedef struct { unsigned char nameLengthA1; unsigned char nameLengthA2; unsigned char nameLengthA3; unsigned char nameLengthA4; unsigned char valueLengthB1; unsigned char valueLengthB2; unsigned char valueLengthB3; unsigned char valueLengthB4; unsigned char nameData unsigned char valueData; }; //    4 
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

In our case, both the name and meanings are short and fit the first option, so we will consider it.

Encode our variables according to the format

 $keyValueFcgiString = ''; foreach ($env as $key => $value) { //        //  128         $keyLen = strlen($key); $lenKeyChar = $keyLen < 128 ? chr($keyLen) : pack('N', $keyLen); $valLen = strlen($value); $valLenChar = $valLen < 128 ? chr($valLen) : pack('N', $valLen); $keyValueFcgiString .= $lenKeyChar . $valLenChar . $key . $value; }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Here values ​​less than 128 bits are encoded by the chr ($ keyLen) function, more than pack ('N', $ valLen) , where 'N' stands for 4 bytes. And then all this is stuck together in one line in accordance with the format of the structure. The body of the recording is ready.

In the header of the record, we transfer everything the same as in the previous record, except for the type (it will be FCGI_PARAMS = 4) and the data length (it will be equal to the length of the name => value pairs, or to the length of the string $ keyValueFcgiString that we formed earlier).

 //  socket_write($socket, pack('CCnnCx', 1, 4, 1, strlen($keyValueFcgiString), 0)); // body socket_write($socket, $keyValueFcgiString); //             //  body socket_write($socket, pack('CCnnCx', 1, 4, 1, 0, 0));
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Getting a response from FCGI_PARAMS

Actually, after all the previous has been done, and everything that it expects has been sent to the application, it starts working and we can only take the result of this work from the socket.

Remember that in response we get the same notes and we also need to interpret them.

We get the header, it is always 8 bytes (we will receive data by byte)

 $buf = ''; $arrData = []; $len = 8; while ($len) { socket_recv($socket, $buf, 1, MSG_WAITALL); //   1       $arrData[] = $buf; $len--; } //      'CCnnCx' $protocol = unpack('C', $arrData[0]); $type = unpack('C', $arrData[1]); $id = unpack('n', $arrData[2] . $arrData[3]); $dataLen = unpack('n', $arrData[4] . $arrData[5])[1]; //   ,        (unpack  ,    ) $foo = unpack('C', $arrData[6]); var_dump($dataLen); //     
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Now, in accordance with the received response body length, we will do another reading from the socket

 $buf2 = ''; $result = []; while ($dataLen) { socket_recv($socket, $buf2, 1, MSG_WAITALL); $result[] = $buf2; $dataLen--; } var_dump(implode('', $result)); //       socket_close($socket);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Hooray it worked! At last!

What do we have in the answer, if for example in this file

 $url = '/path/to/script.php' //    
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

we will write

 <?php echo "My fcgi script";
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

then in the answer we get as a result

Summary

I won’t write much here, so the long article turned out. I hope she helps someone. And I will give the final script itself, it turned out to be quite small. Of course, he can do quite a bit in this form, and he does not have error handling and all this, but he doesn’t need it, he needs him as an example to show the basics.

 <?php $url = '/path/to/script.php'; $env = [ 'REQUEST_METHOD' => 'GET', 'SCRIPT_FILENAME' => $url, ]; $service_port = 9000; $address = '127.0.0.1'; $socket = socket_create(AF_INET, SOCK_STREAM, SOL_TCP); $result = socket_connect($socket, $address, $service_port); //  //     php-fpm //    ,   (    ), id ,   ,     socket_write($socket, pack('CCnnCx', 1, 1, 1, 8, 0)); //     // ,     socket_write($socket, pack('nCxxxxx', 1, 0)); $keyValueFcgiString = ''; foreach ($env as $key => $value) { //        //  128         $keyLen = strlen($key); $lenKeyChar = $keyLen < 128 ? chr($keyLen) : pack('N', $keyLen); $valLen = strlen($value); $valLenChar = $valLen < 128 ? chr($valLen) : pack('N', $valLen); $keyValueFcgiString .= $lenKeyChar . $valLenChar . $key . $value; } // ,      php-fpm           //      //1- ( ), 4-  (,    - FCGI_PARAMS), id  ( ),    (   -),     socket_write($socket, pack('CCnnCx', 1, 4, 1, strlen($keyValueFcgiString), 0)); //      socket_write($socket, $keyValueFcgiString); //  socket_write($socket, pack('CCnnCx', 1, 4, 1, 0, 0)); $buf = ''; $arrData = []; $len = 8; while ($len) { socket_recv($socket, $buf, 1, MSG_WAITALL); //   1       $arrData[] = $buf; $len--; } //      'CCnnCx' $protocol = unpack('C', $arrData[0]); $type = unpack('C', $arrData[1]); $id = unpack('n', $arrData[2] . $arrData[3]); $dataLen = unpack('n', $arrData[4] . $arrData[5])[1]; //   ,        (unpack  ,    ) $foo = unpack('C', $arrData[6]); $buf2 = ''; $result = []; while ($dataLen) { socket_recv($socket, $buf2, 1, MSG_WAITALL); $result[] = $buf2; $dataLen--; } var_dump(implode('', $result)); //       socket_close($socket);