When 'a' is not equal to 'a'. In the wake of one hack
An unpleasant story happened to one of my friends. But how unpleasant she was for Michael is just as entertaining for me.
I must say that my friend is quite a UNIX user: he can install the system himself, install mysql , php and make the simplest nginx settings.
And he has a dozen and a half sites dedicated to construction tools.
One of these sites dedicated to chainsaws is sitting tightly in the TOP search engines. This site is a nonprofit reviewer, but someone has gotten used to attacking him across the throat. Either DDoS , then brute force, then comments will write obscene and send abuses to the hosting and to the ILV.
Suddenly, everything calmed down and this lull was not good, and the site began to gradually leave the top lines of the results.
That was a saying, then the admin bike itself.
The time was approaching sleep when the phone rang: “Sanya, you will not look at my server? It seems to me that they hacked me, I can’t prove it, but the feeling has not left the third week. Maybe it's just time for me to treat paranoia? ”
Next came a half-hour discussion that can be summarized as follows:
Regarding the last paragraph.
Only the white IP frontend is looking into the world. There is no exchange between backends and frontends except http (s), users / passwords are different, keys are not exchanged. On gray addresses, all ports except 80/443 are closed. White IP backends are known only to two users whom Michael fully trusts.
Debian 9 is installed on the front end and by the time of the call the system is isolated from the world by an external firewall and stopped.
“Ok, give me access,” I decide to postpone sleep for an hour. “I'll see with my own eyes.”
Hereinafter:
I start the server, first in rescue-mode . I mount disks, flip through auth- logs, history , system logs, etc., if possible, check the file creation dates, although I understand that a normal cracker would “sweat” behind him, and Misha already knowably “trampled” while searching for himself.
I start in normal mode, especially not yet knowing what to look for, I'm studying configs. He is primarily interested in nginx since, in general, there is nothing other than him on the frontend.
The configs are small, well-structured into a dozen files, I just look through them in a cat ' queue. Everything seems to be clean, but you never missed some include , I'll do a full listing:
I didn’t understand: “Where is the listing?”
The second is added to the listing question: “Why such an ancient version of nginx?”
In addition, the system believes that the version is installed fresh:
I'm calling:
- Misha, why are you rebuilding nginx ?
- Okst, I don’t even know how to do it!
- Ok, well, sleep ...
Nginx is unambiguously reassembled and the listing output for "-T" is hidden for a reason. There is no doubt about hacking and you can just accept it and (since Misha replaced the server with a new one anyway), consider the problem solved.
And indeed, since someone got root privileges , it makes sense to do only system reinstall , and look for something that was poorly consumed there, but this time curiosity defeated the dream. How to find out what they wanted to hide from us?
Let's try to trace:
We look through the trace is clearly not enough lines a la
For fun, compare the findings
I think that part of the code /src/core/nginx.c
was reduced to the form:
or
therefore, listing by -T is not displayed.
If my idea is correct and the problem is only in the variable ngx_dump_config, we will try to install it using gdb , since the key --with-cc-opt -g is present and we hope that the optimization -O2 will not hurt us. At the same time, since I do not know how ngx_dump_config could be processed in case 'T':, we will not call this block, but install it using case 't':
In steps:
As you can see, the real config is different from ours, we select a spurious piece from it:
Let us consider in order what is happening here.
Defined by User-Agent 's yandex / google:
Wordpress service pages are excluded :
And for those who fall under both of the above conditions
in the text of the html- page changes 'o' to 'o' and 'a' to 'a' :
Exactly so, the subtlety is that 'a'! = 'A' is the same as 'o'! = 'O' :
Thus, search engine bots receive, instead of the normal 100% cyrillic text, modified garbage diluted with Latin 'a' and 'o' . I do not presume to discuss how this affects SEO, but it is unlikely that such a literal hash will positively affect positions in the SERP.
What say guys with fantasy.
Debugging with GDB
gdb (1) - Linux man page
strace (1) - Linux man page
Nginx - Module ngx_http_sub_module
About saws, chainsaws and power saws
I must say that my friend is quite a UNIX user: he can install the system himself, install mysql , php and make the simplest nginx settings.
And he has a dozen and a half sites dedicated to construction tools.
One of these sites dedicated to chainsaws is sitting tightly in the TOP search engines. This site is a nonprofit reviewer, but someone has gotten used to attacking him across the throat. Either DDoS , then brute force, then comments will write obscene and send abuses to the hosting and to the ILV.
Suddenly, everything calmed down and this lull was not good, and the site began to gradually leave the top lines of the results.
That was a saying, then the admin bike itself.
The time was approaching sleep when the phone rang: “Sanya, you will not look at my server? It seems to me that they hacked me, I can’t prove it, but the feeling has not left the third week. Maybe it's just time for me to treat paranoia? ”
Next came a half-hour discussion that can be summarized as follows:
- the breeding ground was quite fertile;
- cracker could gain superuser rights;
- the attack (if it took place) was targeted specifically to this site;
- problem areas are fixed and you only need to understand if there was a fact of penetration;
- hacking could not touch the site code and databases.
Regarding the last paragraph.
Only the white IP frontend is looking into the world. There is no exchange between backends and frontends except http (s), users / passwords are different, keys are not exchanged. On gray addresses, all ports except 80/443 are closed. White IP backends are known only to two users whom Michael fully trusts.
Debian 9 is installed on the front end and by the time of the call the system is isolated from the world by an external firewall and stopped.
“Ok, give me access,” I decide to postpone sleep for an hour. “I'll see with my own eyes.”
Hereinafter:
$ grep -F PRETTY_NAME /etc/*releas* PRETTY_NAME="Debian GNU/Linux 9 (stretch)" $ `echo $SHELL` --version GNU bash, version 4.4.12(1)-release (x86_64-pc-linux-gnu) $ nginx -v nginx version: nginx/1.10.3 $ gdb --version GNU gdb (Debian 8.2.1-2) 8.2.1
In search of a possible hack
I start the server, first in rescue-mode . I mount disks, flip through auth- logs, history , system logs, etc., if possible, check the file creation dates, although I understand that a normal cracker would “sweat” behind him, and Misha already knowably “trampled” while searching for himself.
I start in normal mode, especially not yet knowing what to look for, I'm studying configs. He is primarily interested in nginx since, in general, there is nothing other than him on the frontend.
The configs are small, well-structured into a dozen files, I just look through them in a cat ' queue. Everything seems to be clean, but you never missed some include , I'll do a full listing:
$ nginx -T nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
I didn’t understand: “Where is the listing?”
$ nginx -V nginx version: nginx/1.10.3 TLS SNI support enabled configure arguments: --with-cc-opt='-g -O2' --with-ld-opt='-Wl,-z,relro -Wl,-z,now' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_sub_module --with-stream=dynamic --with-stream_ssl_module --with-mail=dynamic --with-mail_ssl_module
The second is added to the listing question: “Why such an ancient version of nginx?”
In addition, the system believes that the version is installed fresh:
$ dpkg -l nginx | grep "[n]ginx" ii nginx 1.14.2-2+deb10u1 all small, powerful, scalable web/proxy server
I'm calling:
- Misha, why are you rebuilding nginx ?
- Okst, I don’t even know how to do it!
- Ok, well, sleep ...
Nginx is unambiguously reassembled and the listing output for "-T" is hidden for a reason. There is no doubt about hacking and you can just accept it and (since Misha replaced the server with a new one anyway), consider the problem solved.
And indeed, since someone got root privileges , it makes sense to do only system reinstall , and look for something that was poorly consumed there, but this time curiosity defeated the dream. How to find out what they wanted to hide from us?
Let's try to trace:
$ strace nginx -T
We look through the trace is clearly not enough lines a la
write(1, "/etc/nginx/nginx.conf", 21/etc/nginx/nginx.conf) = 21 write(1, "... write(1, "\n", 1
For fun, compare the findings
$ strace nginx -T 2>&1 | wc -l 264 $ strace nginx -t 2>&1 | wc -l 264
I think that part of the code /src/core/nginx.c
case 't': ngx_test_config = 1; break; case 'T': ngx_test_config = 1; ngx_dump_config = 1; break;
was reduced to the form:
case 't': ngx_test_config = 1; break; case 'T': ngx_test_config = 1; //ngx_dump_config = 1; break;
or
case 't': ngx_test_config = 1; break; case 'T': ngx_test_config = 1; ngx_dump_config = 0; break;
therefore, listing by -T is not displayed.
But how to see our config?
If my idea is correct and the problem is only in the variable ngx_dump_config, we will try to install it using gdb , since the key --with-cc-opt -g is present and we hope that the optimization -O2 will not hurt us. At the same time, since I do not know how ngx_dump_config could be processed in case 'T':, we will not call this block, but install it using case 't':
Why can I use '-t' along with '-T'
Processing the if (ngx_dump_config) block occurs inside if (ngx_test_config) :
Of course, if the code is changed in this part, and not in case 'T':, then my method will not work.
if (ngx_test_config) { if (!ngx_quiet_mode) { ngx_log_stderr(0, "configuration file %s test is successful", cycle->conf_file.data); } if (ngx_dump_config) { cd = cycle->config_dump.elts; for (i = 0; i < cycle->config_dump.nelts; i++) { ngx_write_stdout("# configuration file "); (void) ngx_write_fd(ngx_stdout, cd[i].name.data, cd[i].name.len); ngx_write_stdout(":" NGX_LINEFEED); b = cd[i].buffer; (void) ngx_write_fd(ngx_stdout, b->pos, b->last - b->pos); ngx_write_stdout(NGX_LINEFEED); } } return 0; }
Of course, if the code is changed in this part, and not in case 'T':, then my method will not work.
Test nginx.conf
Already having solved the problem empirically, it was found that for the operation of the malware a minimal nginx config of the form is needed:
We will use it for brevity in the article.
events { } http { include /etc/nginx/sites-enabled/*; }
We will use it for brevity in the article.
Run the debugger
$ gdb --silent --args nginx -t Reading symbols from nginx...done. (gdb) break main Breakpoint 1 at 0x1f390: file src/core/nginx.c, line 188. (gdb) run Starting program: nginx -t [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Breakpoint 1, main (argc=2, argv=0x7fffffffebc8) at src/core/nginx.c:188 188 src/core/nginx.c: No such file or directory. (gdb) print ngx_dump_config=1 $1 = 1 (gdb) continue Continuing. nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful # configuration file /etc/nginx/nginx.conf: events { } http { map $http_user_agent $sign_user_agent { "~*yandex.com/bots" 1; "~*www.google.com/bot.html" 1; default 0; } map $uri $sign_uri { "~*/wp-" 1; default 0; } map :$sign_user_agent:$sign_uri $sign_o { :1:0 o; default ; } map :$sign_user_agent:$sign_uri $sign_a { :1:0 a; default ; } sub_filter_once off; sub_filter '' $sign_o; sub_filter '' $sign_a; include /etc/nginx/sites-enabled/*; } # configuration file /etc/nginx/sites-enabled/default: [Inferior 1 (process 32581) exited normally] (gdb) quit
In steps:
- set a breakpoint in the main () function
- run the program
- change the value of the variable defining the output of the config ngx_dump_config = 1
- continue / end the program
As you can see, the real config is different from ours, we select a spurious piece from it:
map $http_user_agent $sign_user_agent { "~*yandex.com/bots" 1; "~*www.google.com/bot.html" 1; default 0; } map $uri $sign_uri { "~*/wp-" 1; default 0; } map :$sign_user_agent:$sign_uri $sign_o { :1:0 o; default ; } map :$sign_user_agent:$sign_uri $sign_a { :1:0 a; default ; } sub_filter_once off; sub_filter '' $sign_o; sub_filter '' $sign_a;
Let us consider in order what is happening here.
Defined by User-Agent 's yandex / google:
map $http_user_agent $sign_user_agent { "~*yandex.com/bots" 1; "~*www.google.com/bot.html" 1; default 0; }
Wordpress service pages are excluded :
map $uri $sign_uri { "~*/wp-" 1; default 0; }
And for those who fall under both of the above conditions
map :$sign_user_agent:$sign_uri $sign_o { :1:0 o; default ; } map :$sign_user_agent:$sign_uri $sign_a { :1:0 a; default ; }
in the text of the html- page changes 'o' to 'o' and 'a' to 'a' :
sub_filter_once off; sub_filter '' $sign_o; sub_filter '' $sign_a;
Exactly so, the subtlety is that 'a'! = 'A' is the same as 'o'! = 'O' :
Thus, search engine bots receive, instead of the normal 100% cyrillic text, modified garbage diluted with Latin 'a' and 'o' . I do not presume to discuss how this affects SEO, but it is unlikely that such a literal hash will positively affect positions in the SERP.
What say guys with fantasy.
References
Debugging with GDB
gdb (1) - Linux man page
strace (1) - Linux man page
Nginx - Module ngx_http_sub_module
About saws, chainsaws and power saws
All Articles