Analysis of commits and pull requests in Travis CI, Buddy and AppVeyor using PVS-Studio

Starting with version 7.04, the PVS-Studio analyzer for C and C ++ languages ​​on Linux and macOS has a test ability to check the list of specified files. Using the new mode, you can configure the analyzer to check commits and pull requests. This article will show you how to configure checking the list of modified files of a GitHub project in such popular CI (Continuous Integration) systems as Travis CI, Buddy and AppVeyor.

File List Check Mode

PVS-Studio is a tool for detecting errors and potential vulnerabilities in the source code of programs written in C, C ++, C # and Java. It works on 64-bit systems on Windows, Linux and macOS.



In PVS-Studio version 7.04 for Linux and macOS, the mode of checking the list of source files appeared. This works for projects whose build system allows the compile_commands.json file to be generated. It is needed in order for the analyzer to extract information about the compilation of the specified files. If your build system does not support the generation of the compile_commands.json file, you can try to generate such a file using the Bear utility.



Also, the file list check mode can be used together with the strace trace log of compiler starts (pvs-studio-analyzer trace). To do this, you will first need to carry out a complete assembly of the project and track it so that the analyzer collects complete information about the compilation parameters of all the tested files.



However, this option has a significant drawback - you will either need to make a complete trace of the assembly of the entire project at each start, which in itself contradicts the idea of ​​quickly checking the commit. Or, if the trace result itself is cached, subsequent analyzer starts may not be complete if the dependency structure of the source files changes after the trace (for example, a new #include is added to one of the source files).



Therefore, we do not recommend using the file list check mode with the trace log to check commits or pull requests. In case you can do incremental assembly when checking a commit, consider using the incremental analysis mode.



The list of source files for analysis is saved in a text file and transferred to the analyzer using the -S parameter:

pvs-studio-analyzer analyze ... -f build/compile_commands.json -S check-list.txt
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

In this file, relative or absolute paths to the files are indicated, with each new file must be on a new line. It is permissible to specify not only file names for analysis, but also various text. The analyzer will see that this is not a file, and will ignore the line. This can be useful for commenting if files are specified manually. However, often a list of files will be generated during analysis in CI, for example, these can be files from a commit or pull request.



Now, using this mode, you can quickly check new code before it gets into the main development branch. In order for the verification system to respond to analyzer warnings, the flag --indicate-warnings has been added to the plog-converter utility :

 plog-converter ... --indicate-warnings ... -o /path/to/report.tasks ...
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

With this flag, the converter will return a nonzero code if there are warnings in the analyzer report. Using the return code, you can block the pre-commit hook, commit or pull request, and display the generated analyzer report on the screen, share or send by mail.



Note. The first time you run file list analysis, the entire project will be analyzed, because the analyzer needs to generate a file of dependencies of the project source files from the header files. This is a feature of the analysis of C and C ++ files. In the future, the dependency file can be cached and it will be updated automatically by the analyzer. The advantage of checking commits when using the file list verification mode over using the incremental analysis mode is that you only need to cache this file, not the object files.

General principles of pull request analysis

Analysis of the whole project takes a lot of time, so it makes sense to check only a certain part of it. The problem is that you need to separate the new files from the rest of the project files.



Consider an example of a commit tree with two branches:

Let's imagine that commit A1 contains a fairly large amount of code that has already been tested. A little earlier, we made a branch from the commit A1 and changed some files.



Of course, you noticed that after A1 there were two more commits, but these were also mergers of other branches, because we don’t commit in master . And now the time has come when hotfix is ready. Therefore, a pull request for the merger of B3 and A3 appeared .



Of course, one could check the whole result of their merger, but this would be too long and unjustified, since only a few files were changed. Therefore, it is more efficient to analyze only the changed ones.



To do this, we get the difference between the branches, being in the HEAD of the branch from which we want to merge into master:

 git diff --name-only HEAD origin/$MERGE_BASE > .pvs-pr.list
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

$ MERGE_BASE we will consider in detail later. The fact is that not every CI service provides the necessary information about the basis for merging, so every time you have to come up with new ways to get this data. This will be detailed below in each of the web services described.



So, we got the difference between the branches, or rather, a list of file names that have been changed. Now we need to give the .pvs-pr.list file (we redirected the output to it above) to the analyzer:

 pvs-studio-analyzer analyze -j8 \ -o PVS-Studio.log \ -S .pvs-pr.list
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

After analysis, we need to convert the log file (PVS-Studio.log) into a format that is easy to read:

 plog-converter -t errorfile PVS-Studio.log --cerr -w
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

This command will list errors in stderr (standard error message output stream).



Only now we need to not only display errors, but also inform our service for assembly and testing of problems. To do this, the -W flag ( --indicate-warnings ) was added to the converter. If there is at least one analyzer warning, the return code of the plog-converter utility will change to 2, which, in turn, will inform the CI service about the presence of potential errors in the pull request files.

Travis ci

The configuration is in the form of a .travis.yml file. For convenience, I advise you to put everything in a separate bash script with functions that will be called from the .travis.yml file ( bash script_name.sh function_name ).



We will add the necessary code to the bash script, so we get more functionality. In the install section, write the following:

 install: - bash .travis.sh travis_install
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

If you had any instructions, you can transfer them to the script by removing hyphens.



Open the .travis.sh file and add the analyzer installation to the travis_install () function:

 travis_install() { wget -q -O - https://files.viva64.com/etc/pubkey.txt \ | sudo apt-key add - sudo wget -O /etc/apt/sources.list.d/viva64.list \ https://files.viva64.com/etc/viva64.list sudo apt-get update -qq sudo apt-get install -qq pvs-studio }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Now add the analysis run to the script section:

 script: - bash .travis.sh travis_script
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

And in the bash script:

 travis_script() { pvs-studio-analyzer credentials $PVS_USERNAME $PVS_KEY if [ "$TRAVIS_PULL_REQUEST" != "false" ]; then git diff --name-only origin/HEAD > .pvs-pr.list pvs-studio-analyzer analyze -j8 \ -o PVS-Studio.log \ -S .pvs-pr.list \ --disableLicenseExpirationCheck else pvs-studio-analyzer analyze -j8 \ -o PVS-Studio.log \ --disableLicenseExpirationCheck fi plog-converter -t errorfile PVS-Studio.log --cerr -w }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

This code needs to be run after building the project, for example, if you had a build on CMake:

 travis_script() { CMAKE_ARGS="-DCMAKE_EXPORT_COMPILE_COMMANDS=On ${CMAKE_ARGS}" cmake $CMAKE_ARGS CMakeLists.txt make -j8 }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

It turns out like this:

 travis_script() { CMAKE_ARGS="-DCMAKE_EXPORT_COMPILE_COMMANDS=On ${CMAKE_ARGS}" cmake $CMAKE_ARGS CMakeLists.txt make -j8 pvs-studio-analyzer credentials $PVS_USERNAME $PVS_KEY if [ "$TRAVIS_PULL_REQUEST" != "false" ]; then git diff --name-only origin/HEAD > .pvs-pr.list pvs-studio-analyzer analyze -j8 \ -o PVS-Studio.log \ -S .pvs-pr.list \ --disableLicenseExpirationCheck else pvs-studio-analyzer analyze -j8 \ -o PVS-Studio.log \ --disableLicenseExpirationCheck fi plog-converter -t errorfile PVS-Studio.log --cerr -w }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

You have probably already noticed the indicated environment variables $ TRAVIS_PULL_REQUEST and $ TRAVIS_BRANCH . Travis CI announces them on its own:

• $ TRAVIS_PULL_REQUEST stores the pull request number or false if it is a regular branch;
• $ TRAVIS_REPO_SLUG stores the name of the project repository.

The algorithm of this function:

Travis CI responds to return codes, so the presence of warnings will tell the service to mark the commit as containing errors.



Now, let's take a closer look at this line of code:

 git diff --name-only origin/HEAD > .pvs-pr.list
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

The fact is that Travis CI automatically merges branches during pull request analysis:

Therefore, we analyze A4 , not B3-> A3 . Because of this feature, we need to calculate the difference with A3 , which is precisely the top of the branch from origin .



One important detail remains - caching dependencies of header files on compiled translation units (* .c, * .cc, * .cpp, etc.). The analyzer calculates these dependencies at the first start in the file list check mode and then saves it in the .PVS-Studio directory. Travis CI allows you to cache folders, so we will save the data of the .PVS-Studio / directory:

 cache: directories: - .PVS-Studio/
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

This code needs to be added to the .travis.yml file. This directory stores various data collected after analysis, which will significantly speed up subsequent launches of file list analysis or incremental analysis. If this is not done, then the analyzer will actually analyze all the files every time.

Buddy

Like Travis CI, Buddy provides the ability to automatically build and test projects that are stored on GitHub. Unlike Travis CI, it is configured in the web interface (bash support is available), so there is no need to store configuration files in the project.



First of all, we need to add a new action to the assembly line:

We indicate the compiler that was used to build the project. Pay attention to the docker container that is installed in this action. For example, there is a special container for GCC:

Now install PVS-Studio and the necessary utilities:

Add the following lines to the editor:

 apt-get update && apt-get -y install wget gnupg jq wget -q -O - https://files.viva64.com/etc/pubkey.txt | apt-key add - wget -O /etc/apt/sources.list.d/viva64.list \ https://files.viva64.com/etc/viva64.list apt-get update && apt-get -y install pvs-studio
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Now go to the Run tab (first icon) and add the following code to the appropriate field of the editor:

 pvs-studio-analyzer credentials $PVS_USERNAME $PVS_KEY if [ "$BUDDY_EXECUTION_PULL_REQUEST_NO" != '' ]; then PULL_REQUEST_ID="pulls/$BUDDY_EXECUTION_PULL_REQUEST_NO" MERGE_BASE=`wget -qO - \ https://api.github.com/repos/${BUDDY_REPO_SLUG}/${PULL_REQUEST_ID} \ | jq -r ".base.ref"` git diff --name-only HEAD origin/$MERGE_BASE > .pvs-pr.list pvs-studio-analyzer analyze -j8 \ -o PVS-Studio.log \ --disableLicenseExpirationCheck \ -S .pvs-pr.list else pvs-studio-analyzer analyze -j8 \ -o PVS-Studio.log \ --disableLicenseExpirationCheck fi plog-converter -t errorfile PVS-Studio.log --cerr -w
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

If you read the section on Travs-CI, then this code is already familiar to you, however, now a new stage has appeared:

The fact is that now we are analyzing not the result of the merger, but the HEAD of the branch from which the pull request is made:

Therefore, we are in the conditional commit B3 and we need to get the difference with A3 :

 PULL_REQUEST_ID="pulls/$BUDDY_EXECUTION_PULL_REQUEST_NO" MERGE_BASE=`wget -qO - \ https://api.github.com/repos/${BUDDY_REPO_SLUG}/${PULL_REQUEST_ID} \ | jq -r ".base.ref"` git diff --name-only HEAD origin/$MERGE_BASE > .pvs-pr.list
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

To determine A3, use the GitHub API:

 https://api.github.com/repos/${USERNAME}/${REPO}/pulls/${PULL_REQUEST_ID}
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

We used the following variables that Buddy provides:

• $ BUDDY_EXECUTION_PULL_REQEUST_NO - pull request number;
• $ BUDDY_REPO_SLUG - a combination of a username and a repository (for example max / test).

Now save the changes using the button below and enable pull request analysis:

Unlike Travis CI, we do not need to specify .pvs-studio for caching, since Buddy automatically caches all files for subsequent launches. Therefore, the last thing left is to save the login and password for PVS-Studio in Buddy. After saving the changes, we will go back to Pipeline. We need to go on to setting up the variables and add the login and key for PVS-Studio:

After that, the appearance of a new pull request or commit will trigger a check. If the commit contains errors, then Buddy will indicate this on the pull request page.

Appveyor

AppVeyor setup is similar to Buddy, since everything happens in the web interface and there is no need to add a * .yml file to the project repository.



Go to the Settings tab in the project overview:

Scroll this page down and turn on cache saving to build pull requests:

Now go to the Environment tab, where we specify the image for the assembly and the necessary environment variables:

If you read the previous sections, you are well acquainted with these two variables - PVS_KEY and PVS_USERNAME . If not, then I remind you that they are necessary for checking the license of the PVS-Studio analyzer. In the future, we will meet them again in Bash scripts.



On the same page below we indicate the folder for caching:

If we do not, then we will analyze the entire project instead of a couple of files, but we will get the output from the specified files. Therefore, it is important to enter the correct directory name.



Now it's time for the script to check. Open the Tests tab and select Script:

Insert the following code into this form:

 sudo apt-get update && sudo apt-get -y install jq wget -q -O - https://files.viva64.com/etc/pubkey.txt \ | sudo apt-key add - sudo wget -O /etc/apt/sources.list.d/viva64.list \ https://files.viva64.com/etc/viva64.list sudo apt-get update && sudo apt-get -y install pvs-studio pvs-studio-analyzer credentials $PVS_USERNAME $PVS_KEY PWD=$(pwd -L) if [ "$APPVEYOR_PULL_REQUEST_NUMBER" != '' ]; then PULL_REQUEST_ID="pulls/$APPVEYOR_PULL_REQUEST_NUMBER" MERGE_BASE=`wget -qO - \ https://api.github.com/repos/${APPVEYOR_REPO_NAME}/${PULL_REQUEST_ID} \ | jq -r ".base.ref"` git diff --name-only HEAD origin/$MERGE_BASE > .pvs-pr.list pvs-studio-analyzer analyze -j8 \ -o PVS-Studio.log \ --disableLicenseExpirationCheck \ --dump-files --dump-log pvs-dump.log \ -S .pvs-pr.list else pvs-studio-analyzer analyze -j8 \ -o PVS-Studio.log \ --disableLicenseExpirationCheck fi plog-converter -t errorfile PVS-Studio.log --cerr -w
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Pay attention to the following part of the code:

 PWD=$(pwd -L) if [ "$APPVEYOR_PULL_REQUEST_NUMBER" != '' ]; then PULL_REQUEST_ID="pulls/$APPVEYOR_PULL_REQUEST_NUMBER" MERGE_BASE=`wget -qO - \ https://api.github.com/repos/${APPVEYOR_REPO_NAME}/${PULL_REQUEST_ID} \ | jq -r ".base.ref"` git diff --name-only HEAD origin/$MERGE_BASE > .pvs-pr.list pvs-studio-analyzer analyze -j8 \ -o PVS-Studio.log \ --disableLicenseExpirationCheck \ --dump-files --dump-log pvs-dump.log \ -S .pvs-pr.list else pvs-studio-analyzer analyze -j8 \ -o PVS-Studio.log \ --disableLicenseExpirationCheck fi
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

The rather specific assignment of the value of the pwd command to the variable that should store this default value seems strange at first glance, however, I'll explain everything now.



While configuring the analyzer in AppVeyor, I encountered extremely strange analyzer behavior. On the one hand, everything worked correctly, but the analysis did not start. I spent a lot of time to notice that we are in the / home / appveyor / projects / testcalc / directory, and the analyzer is sure that we are in / opt / appveyor / build-agent /. Then I realized that the $ PWD variable is lying a little. For this reason, I manually updated its value before starting the analysis.



And then everything, as before:

Now consider the following snippet:

 PULL_REQUEST_ID="pulls/$APPVEYOR_PULL_REQUEST_NUMBER" MERGE_BASE=`wget -qO - \ https://api.github.com/repos/${APPVEYOR_REPO_NAME}/${PULL_REQUEST_ID} \ | jq -r ".base.ref"`
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

In it, we get the difference between the branches over which the pull request is declared. To do this, we need the following environment variables:

• $ APPVEYOR_PULL_REQUEST_NUMBER - number of pull request;
• $ APPVEYOR_REPO_NAME - username and project repository.

Conclusion

Of course, we did not consider all of the possible services of continuous integration, however, all of them have very similar to each other specifics of work. With the exception of caching, each service makes its own “bike”, so everything is always different.



Somewhere, like in Travis-CI, a couple lines of code and caching works flawlessly; somewhere, like in AppVeyor, you just need to specify the folder in the settings; but somewhere you need to create unique keys and try to convince the system to give you the opportunity to overwrite the cached fragment. Therefore, if you want to configure the analysis of pull requests on the continuous integration service, which was not discussed above, first make sure that you will not have problems with caching.



Thanks for your attention. If something doesn’t work out, feel free to write to us in support . We will prompt and help.

If you want to share this article with an English-speaking audience, then please use the link to the translation: Maxim Zvyagintsev. Analysis of commits and pull requests in Travis CI, Buddy and AppVeyor using PVS-Studio .