🖐🏻 🙇 👸🏿 Building Dynamic Modules for Nginx 👻 👨‍👦 😵

Recently, we built a dynamic module for Nginx, and when everything was ready, it turned out that our module was not compatible with Nginx, which was already installed on the server. We could not find a ready-made solution to the problem and we began to fight it on our own. We spent a lot of time, but got a new experience and, most importantly, a working solution. Which I would like to share.

Let's start with a description of the dynamic module assembly process using the example https://github.com/vozlt/nginx-module-vts . And then we will show what error occurs and what to do with it.

Inputs:

Debian 9 OS

Nginx v1.10.3 from the standard Debian repository. nginx -V

output:

 nginx version: nginx/1.10.3 built with OpenSSL 1.1.0k 28 May 2019 (running with OpenSSL 1.1.1c 28 May 2019) TLS SNI support enabled configure arguments: --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-DhOtPd/nginx-1.10.3=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-z,relro -Wl,-z,now' --prefix=/usr/share/nginx --conf-path=/etc/ nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path =/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module - -with-http_stub_status_module --with-http_realip_mod ule --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module= dynamic --with-http_sub_module --with-http_xslt_module=dynamic --with-stream=dynamic --with-stream_ssl_module --with-mail=dynamic --with-mail_ssl_module --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-auth-pam --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-dav-ext-module --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-echo --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-upstream-fair --add-dynamic-module=/build/nginx-DhOtPd/nginx -1.10.3/debian/modules/ngx_http_substitutions_filter_module

The output of this command contains information that is necessary to build dynamic modules, namely, everything that is written after configure arguments:

and before the first - --add-dynamic-module

, that is:

 --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-DhOtPd/nginx-1.10.3=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-z,relro -Wl,-z,now' --prefix=/usr/share/nginx --conf-path=/etc/ nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path =/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_mod ule --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module= dynamic --with-http_sub_module --with-http_xslt_module=dynamic --with-stream=dynamic --with-stream_ssl_module --with-mail=dynamic --with-mail_ssl_module

So, we proceed to the assembly of the module:

For assembly, it is convenient to use a docker container so as not to clutter up the main system, take the debian: 9 image as the basis, for the convenience of copying the finished module from the container, you can specify volume. The command to start the container in this case will look like this:

 docker run --rm -it -v /tmp/nginx_module:/nginx_module debian:9 bash

After starting the container, install the necessary packages. Please note that the packages are selected according to the output of the nginx -V command, so you may not need some libraries. In any case, the configure script will warn you about the absence of dependencies:

 apt update apt install git make gcc autoconf wget libpcre3-dev libpcre++-dev zlib1g-dev libxml2-dev libxslt-dev libgd-dev libgeoip-dev

The libssl-dev package is missing from this list, because the nginx installed on the server is built with the version 1.1.0k of OpenSSL, and at the time of writing when installing the libssl-dev package from the repository, the version of OpenSSL in it was already 1.1.0l. Therefore, the OpenSSL source code for the correct version must be downloaded separately.

Download the desired version from the website http://nginx.org/download nginx, in our case it is 1.10.3. Download the OpenSSL sources from the official site https://www.openssl.org/source/old and, in fact, download the source code of the nginx-module-vts module from github.

 cd /usr/local/src/ wget http://nginx.org/download/nginx-1.10.3.tar.gz wget https://www.openssl.org/source/old/1.1.0/openssl-1.1.0k.tar.gz git clone git://github.com/vozlt/nginx-module-vts.git tar xvfz nginx-1.10.3.tar.gz tar xzvf openssl-1.1.0k.tar.gz cd nginx-1.10.3

Now you are ready to run the configure script. We specify all parameters from the output of the nginx -V command as the script parameters, which I wrote about at the beginning of the article. We also add the --add-dynamic-module parameter to build the nginx-module-vts module and specify the path to the directory with the OpenSSL source files through the --with-openssl parameter. The final command will look like this:

 ./configure --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-DhOtPd/nginx-1.10.3=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-z,relro -Wl,-z,now' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_sub_module --with-http_xslt_module=dynamic --with-stream=dynamic --with-stream_ssl_module --with-mail=dynamic --with-mail_ssl_module --with-openssl=../openssl-1.1.0k/ --add-dynamic-module=../nginx-module-vts/

After the configure

script finishes working, we start the assembly of modules; we don’t need to collect all nginx

:

 make modules

Note that this command is only available with nginx

version 1.9.13.

After the assembly process is completed, copy the resulting so file to volume

, which was mounted when the container started:

 cp objs/ngx_http_vhost_traffic_status_module.so /nginx_module/

Next, we place the file on the target server, in this case, the directory with the modules is located on the path /usr/share/nginx/modules

. In order to enable the module in nginx

we create the file /etc/nginx/modules-enabled/mod-http-vhost-traffic-status.conf

with the following contents:

 load_module modules/ngx_http_vhost_traffic_status_module.so;

And create a symbolic link to this file in the /etc/nginx/modules-enabled

directory.

After the work done, we call the nginx -t

command and ... we get the error:

 nginx: [emerg] module "/usr/share/nginx/modules/ngx_http_vhost_traffic_status_module.so" is not binary compatible in /etc/nginx/modules-enabled/50-mod-http-vhost-traffic-status.conf:1

And here, usually, there is some bewilderment, because the source versions and the nginx

version installed on the server are the same, the OpenSSL versions too, the parameters for the configure

script are also copied entirely. So what's the deal?

To understand this problem, I had to understand how nginx checks if the dynamic module is suitable for it. The code that is located in the nginx

sources in the src/core/ngx_module.h

( https://github.com/nginx/nginx/blob/master/src/core/ngx_module.h ) is responsible for this check. There are a number of checks in this file, in this case 34, during which nginx sets for variables of the form NGX_MODULE_SIGNATURE_0

(1,2,3, etc.)

values 1 or 0. Next is the variable NGX_MODULE_SIGNATURE

which collects the results of all checks in one line. Accordingly, the problem is that the signature string of the new module does not match the signature string of the existing nginx. To check the values of signature strings, you can use the following commands:

The signature line of the module that was built:

 strings /usr/share/nginx/modules/ngx_http_vhost_traffic_status_module.so| fgrep '8,4,8' 8,4,8,000011111101011111111111110110111

Signature line of the nginx

binary file installed on the server:

 strings /usr/sbin/nginx| fgrep '8,4,8' 8,4,8,000011111101011111111111110111111

When comparing these lines with the naked eye, it can be seen that in the module signature line, the fourth variable from the end turned out to be 0, while nginx

has 1. It is necessary to refer to the src/core/ngx_module.h

file to understand what exactly is missing in the module src/core/ngx_module.h

and find the fourth variable from the end, it looks something like this:

 #if (NGX_HTTP_REALIP) #define NGX_MODULE_SIGNATURE_29 "1" #else #define NGX_MODULE_SIGNATURE_29 "0" #endif #if (NGX_HTTP_HEADERS) #define NGX_MODULE_SIGNATURE_30 "1" #else #define NGX_MODULE_SIGNATURE_30 "0" #endif #if (NGX_HTTP_DAV) #define NGX_MODULE_SIGNATURE_31 "1" #else #define NGX_MODULE_SIGNATURE_31 "0" #endif #if (NGX_HTTP_CACHE) #define NGX_MODULE_SIGNATURE_32 "1" #else #define NGX_MODULE_SIGNATURE_32 "0" #endif #if (NGX_HTTP_UPSTREAM_ZONE) #define NGX_MODULE_SIGNATURE_33 "1" #else #define NGX_MODULE_SIGNATURE_33 "0" #endif #define NGX_MODULE_SIGNATURE

We are interested in the variable NGX_HTTP_HEADERS

, when building nginx

it was 1, and when building the module 0. In order for the module to NGX_HTTP_HEADERS

with NGX_HTTP_HEADERS

you need to adjust the config file in the directory with the source code of the module by adding the line have=NGX_HTTP_HEADERS . auto/have

have=NGX_HTTP_HEADERS . auto/have

at the beginning of the config.

file config.

The following is the beginning of the config file after making the changes:

 ngx_addon_name=ngx_http_vhost_traffic_status_module have=NGX_STAT_STUB . auto/have have=NGX_HTTP_HEADERS . auto/have ...

Next, do make clean

, and then re-run configure

and make modules

. After assembling the module, we check its signature string and see that now it corresponds to the nginx

string:

 $ strings /usr/sbin/nginx| fgrep '8,4,8' 8,4,8,000011111101011111111111110111111 $ strings /usr/share/nginx/modules/ngx_http_vhost_traffic_status_module.so | fgrep '8,4,8' 8,4,8,000011111101011111111111110111111

After that, the module connects without any problems and you can collect advanced statistics with nginx

.

In your case, perhaps the signature line will look different, as well as the variables that are responsible for each position in this line. However, now you can understand what exactly is wrong with the assembly of the module and fix the problem.

I hope this article saves someone a few hours of time, since I personally several times faced the problem that the module did not fit nginx

, as a result, as a rule, I solved this through the full assembly of nginx

along with the desired module.

Building Dynamic Modules for Nginx

Also read other articles on our blog:

More articles: