The book "Safe DevOps. Efficient system operation ”

Hello, habrozhiteli! An application running in the cloud has many advantages, but at the same time is subject to special threats. The task of DevOps teams is to assess these risks and strengthen the system’s protection against them. The book is based on the unique experience of the author and offers the most important strategic solutions for protecting web applications from attacks, preventing intrusion attempts. You will see how to ensure reliability with automated testing, continuous delivery and key DevOps processes. Learn to identify, evaluate and fix vulnerabilities that exist in your application. The author will help you navigate in cloud configurations, as well as use popular automation tools. Requires Linux knowledge and knowledge of standard DevOps practices such as CI, CD, and unit testing.

Excerpt. Chapter 8. Log Analysis for Intrusion and Attack Detection

In this chapter:

• Exploring analysis-level components in a logging pipeline.
• Detection of intrusions and attacks using string signatures, statistics and historical data.
• Managing ways to optimally notify users.

In Chapter 7, you learned how to create a logging pipeline that collects, transfers, analyzes, and stores logs from the entire infrastructure, and also provides access to them. A multi-level pipeline creates a flexible infrastructure in which logs from various sources are used to monitor the activity of the organization’s services. Chapter 7 provided an overview of the functionality provided by all levels of the pipeline. In this chapter, we will focus on the third level — the analysis level — and dive into techniques and code examples related to detecting intrusions and attacks on services.



The logging pipeline used by Mozilla when writing the book is similar to that shown in chapter 7. The pipeline is used to monitor the state of Firefox clients (called telemetry ) in a natural environment, process application and service logs, and detect unusual activity. The logical center of the pipeline is at the level of analysis, composed of many small programs that are constantly looking for something unusual. These small programs are not so advanced as to deal with the input and output of log events, so they transfer this task to a dedicated data center - the Hindsight program (http://mng.bz/m4gg), designed to perform the work of analyzing plug-ins on data streams .



In this chapter, we will use Hindsight to read various types of magazines and write original plugins to analyze them.

NOTE

Sample logs and plugins for this chapter are located at securing-devops.com/ch08/logging-pipeline . You need to copy this repository to your local machine and get the Hindsight Docker container in order to run the examples.

Let's start by describing how the various parts of the analysis level are arranged: Hindsight is located in the middle, and the collection and storage levels are on both sides. Then let's talk about three different approaches to detect intrusions and attacks. For the simplest of them, string signatures are used that contain information about known attacks to send notifications. Next, we compare the statistical models and the approach with signatures, and also learn how these two approaches complement each other. Finally, let’s take a look at ways to apply historical user activity data to identify suspicious areas among connections.



The last section of the chapter is about sending notifications. It is unlikely that you will want to receive thousands of notifications from the analysis level every day, which will create a lot of noise, instead of being useful. If this is not changed, the recipients will regard the notifications as spam and ignore them. In the final section of this chapter, we will consider the best practices for organizing the sending of notifications, as well as discuss ways to accurately and efficiently send notifications to administrators and end users.

8.2. Detecting attacks using string signatures

When you work with logs, you work with strings. This means that the easiest way to detect signs of fraud is to compare the logs with a list of known malicious lines. It may seem simple, but it was precisely these years that the security industry had been involved in the industry. Web application firewalls (WAFs), so popular in the mid-2000s, were essentially a repository of regular expressions, which were checked against every request sent to a web application.

Do not rely on regular expressions

I once worked for a bank in which this type of protection was widely used. The security team was responsible for supporting WAF, which protected a variety of online services, including a trading service for customers. Each web request sent to this service went through hundreds of regular expressions before getting to the application server. Once a developer from the online trading service team decided to take a look at these regular expressions. I don’t know what prompted the engineer to start reading the contents of the file, filled mainly with slashes, dollar symbols, asterisks, pluses, square and parentheses, but he took up it. And somewhere on the 418th line in a complex regular expression, he found a suspicious combination of “. +”. Two innocent characters that allowed absolutely everything to go on painlessly: this regular expression is similar to the value “allow everything”.



Our pride is the web application firewall for several thousand euros, supported by a whole team, has performed hundreds of regular expression checks, affecting performance and complicating the design of an already complex system, and all in order to skip everything unhindered. Of course, we soon corrected the problem, but my belief in the use of regular expressions to ensure security has not risen since then. If you want to implement this type of protection system in your organization, take into account all its complexity so that this does not happen to you.

When used correctly, regular expressions can become a powerful tool, but they are incredibly difficult to write, maintain over time - even more difficult, and their large-scale implementation is expensive. Take a look at the regular expression ((\% 3C) | <) ((\% 2F) | \ /) * [a-z0-9 \%] + ((\% 3E) |>). You won’t guess what it is used for, so I’ll tell you: with it, you can search for injections in the strings of HTTP requests by detecting the presence of opening and closing characters of the inequality <> and their contents. You will receive this type of HTTP request line from an attacker who is trying to inject malicious JavaScript code into your application to end up with a cross-site scripting attack, which we discussed in Chapter 3.



This regular expression can be used to identify suspicious requests that contain injection attempts. Listing 8.7 shows an example parser that implements this by checking that each of the transmitted NGINX access logs matches the regular expression. The regular expression is stored in the local xss variable, its value is compared with each Fields [request] value using the rex.match () function.



If there are matches, then the add_to_payload () function sends a notification that the output plug-in can receive and transmit to the destination.

Listing 8.7. A plugin that detects log messages containing attack signs in the query string

An example output from this plugin is shown in Listing 8.8. There are generated several notifications by sample magazines, some of which turned out to be false positive. This happened in part because sample logs were artificially created using the ZAP Vulnerability Scanner, and also because query strings did not tend to contain HTML tags. In particular, this regular expression will not have too high a rate for finding false positive matches.

Listing 8.8. Examples of notifications generated by the XSS analysis plugin

This is just one regular expression for one specific type of attack. For this approach to be useful, you need to find something more than regular expressions. For starters, you could collect signatures from various sources and, as more and more suspicious signs are found in the logs, gradually increase the size of your database.



Listing 8.9 shows a modified version of the XSS parser that looks for various signs of an attack (http://mng.bz/62h8). This script shows how you can use Lua tables to store a list of attributes and use it cyclically to analyze incoming events. In this code example, the suspicious_terms table is a simple row of rows that uses substrings to search for regular expressions, which is much faster. The suspicious_terms table uses the key-value format to store labels along with the expression in order to be able to remind you that the expression should find.

Listing 8.9. Search for signs of attacks using strings and expressions

You can run this analyzer using the test settings described at the beginning of the chapter. Launch the Docker container with mounted directories and the analyzer output will be written to output / payload / analysis.suspicious_signatures.alerts.txt. The plugin will send thousands of notifications, which is expected, since these logs are generated by the ZAP vulnerability scanner. This approach can be considered successful, but it has flaws that you should consider.

• Regular expressions are hard to write, and even harder to read . You will make mistakes that are not easy to investigate and which will require hours of painful debugging. There are only four regular expressions in this analyzer, but reading this part of the code is already difficult. No matter how powerful and attractive the tool may be regular expressions, I would not recommend working with them constantly.
• With this approach, too many notifications are generated . Web applications open to the Internet receive a lot of unusual traffic, malicious and not so much. Generating notifications for each unusual sign will drive any security team crazy for several weeks, even if the rate of false positives is low. Abnormal traffic is a natural occurrence for services running on the Internet.

You could fix both of these problems by applying a mathematical approach and making this perfect system for detecting anomalies a little less perfect. In the next section, we will look at how to use statistical methods to send notifications when a threshold is overcome as a way to reduce noise from the anomaly detection logic.

about the author

At the time of writing, Julien Vehen is managing the operations security team at Firefox, Mozilla. He is responsible for creating, implementing, and operating a web services security strategy that millions of Firefox users interact with every day. Julien focused on protecting network services in the early 2000s. He began working as a system administrator in Linux, and in 2007 received a master's degree in information security.



»More details on the book can be found on the publisher’s website

» Contents

» Excerpt



25% off coupon for hawkers - DevOps



Upon payment of the paper version of the book, an electronic book is sent by e-mail.