Recently I read an article on Habré about a very useful tool, and since I had been looking for some kind of project for a long time to start contributing, I decided to see what is on the github and how I can help. One issue was about creating a wrapper (I’ll use wrapper later) for the C-library. At that moment I thought, “Oh, something interesting, I'm sure it will take no more than an hour.” How much I was mistaken.

In this article, I decided to show not one way to solve a similar problem, but several different options. I will show you options for creating modules in Python with compilation in C, using a small self-written library C in Python, and - the last option - using a large C library in Python without pain and pxd files.

Cython

Books have already been written about this, there are many articles, including on Habré, so I won’t focus too much on the installation or some basic things. Read more here

Using Cython, we can solve several problems. For some instances of C code in python, it generally fits perfectly and partially solves the problem with library imports.

Let's look at a simple example from the official documentation.

from __future__ import print_function def fib(n): """Print the Fibonacci series up to n.""" a, b = 0, 1 while b < n: print(b, end=' ') a, b = b, a + b print()
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Save this file as fib.pyx



.

.pyx is a special format of Cython files, which is similar to .c for C code and contains some functionality. There is also .pxd , in C it is .h and contains a description of functions, structures, etc.

In order to somehow interact with the fib function, we need to "compile" the code. To do this, create setup.py



with this content.

 from distutils.core import setup from Cython.Build import cythonize setup( ext_modules=cythonize("fib.pyx"), )
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

After running the python3 setup.py build_ext --inplace



you can import it in regular python and enjoy the speed of work as in normal compiled languages.

 import fib fib.fib(2000)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

But here we wrote Python code and turned it into C, but what about writing C code and running it in Python?

Not a problem. We create a new folder, inside we create the lib



folder in which we will create lib/include



and lib/src



, in fact, everyone who worked with C already knows what will be there. In the main folder, create another python_wrap



folder.

Let's struct.h



to lib/include



and create struct.h



, in which we will describe one function and see how to work with structures in C through Cython.

 typedef struct struct_test{ int a; int b; } struct_test; int minus(struct_test a);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Let's create another file, which we will call include.h



, it will have another function and import the structure from struct.h

 #include "struct.h" int sum(struct_test param_in_struct);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Now we will describe these functions in the file lib/src/test_main.c

 #include "include.h" int sum(struct_test param_in_struct){ return param_in_struct.a+param_in_struct.b; } int minus(struct_test param_in_struct){ return param_in_struct.a-param_in_struct.b; }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Yes, I do not pretend to the originality of variable names, but we have almost finished the C-part. What else? Add a Makefile, or rather CMake. In the lib



folder, create CMakeLists.txt



.

 set (TARGET "mal") include_directories( include src ) set (SOURCES ./src/test_main.c ) set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS}") add_library(${TARGET} SHARED ${SOURCES}) target_link_libraries(${TARGET} ${LINKLIBS}) add_library(${TARGET}static STATIC ${SOURCES}) target_link_libraries(${TARGET}static ${LINKLIBS})
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

From the main directory, we need to indicate that we have a project for compilation in the lib



folder. Create another CMakeLists.txt



file, but already at the root.

 cmake_minimum_required(VERSION 2.8.2 FATAL_ERROR) cmake_policy(VERSION 2.8) project( TEST ) set (CMAKE_C_FLAGS "-Werror -Wall -Wextra -Wno-unused-parameter -D_GNU_SOURCE -std=c11 -O3 -g ${CMAKE_C_FLAGS}") add_custom_target( ReplicatePythonSourceTree ALL ${CMAKE_COMMAND} -P ${CMAKE_CURRENT_SOURCE_DIR}/cmake/ReplicatePythonSourceTree.cmake ${CMAKE_CURRENT_BINARY_DIR} WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} ) include( GNUInstallDirs ) add_subdirectory(lib)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Here I use a small file that transfers the Python wrapper structure to the build directory so that you can compile the Python files. This may not be necessary if you pass relative paths to the include



directory and the place where the library will be. For example, if the library is already compiled and installed in the system, then we will set the paths to the system directories, but more on that later.

 # Note: when executed in the build dir, then CMAKE_CURRENT_SOURCE_DIR is the # build dir. file( COPY setup.py DESTINATION "${CMAKE_ARGV3}" FILES_MATCHING PATTERN "*.py" ) file( COPY lib/src lib/include DESTINATION "${CMAKE_ARGV3}") file(GLOB MY_WRAP "python_wrap/*" ) file( COPY ${MY_WRAP} DESTINATION "${CMAKE_ARGV3}")
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Before assembling our project, let's take a look at the Python part. In the python_wrap



folder python_wrap



create two files main.pxd



and main.pyx



. In main.pxd



we need to describe what we have in *.h



files.

 cdef extern from "include/include.h": ctypedef struct struct_test: int a int b int sum(struct_test param_in_struct); int minus(struct_test param_in_struct);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Using cdef extern from "include/include.h"



, we indicate which file we will describe. Next comes the ctypedef struct struct_test:



description of the structure so that it can be used from Python code. At the end, in fact, a description of two functions. I want to note that we need to describe all include that are in include.h



, if it imports a structure and function from another header file, we believe that all this is in one file.

In main.pyx



we write the functions of transition from Python to C. This is not necessary, but why load Python code with structures for C. To create a structure, it is enough to define a dictionary with all parameters.

 from main cimport sum, minus def sum_py(int x, int y): return sum({"a":x,"b":y}) def minus_py(int x, int y): return minus({"a":x,"b":y})
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Now we need to make it all come together. Add the setup.py



file to the project root, as we did before.

 from distutils.core import setup from distutils.extension import Extension from Cython.Distutils import build_ext ext_modules = [Extension('main', ['main.pyx'], libraries=['mal'], library_dirs=['lib/'])] setup(name = 'work extension module', cmdclass = {'build_ext': build_ext}, ext_modules = ext_modules)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

In order to compile C code and compile our library, we will create a simple bash script.

 #!/bin/sh rm -rf build; mkdir build && cd build cmake .. && make $@ python3 setup.py build_ext -i
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

We launch and check

 $ sh build.sh $ python3 > import build.main as main > dir(main) [.... 'minus_py', 'sum_py'] > main.minus_py(10,2) 8 > main.sum_py(10,2) 12
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Ctypesgen

The previous example was very simple and straightforward, but what if you need to wrap a very large library, write all .pxd files with your hands for a very long time and is difficult, so there is a reasonable question, what can be used to automate the process?

We git clone https://github.com/davidjamesca/ctypesgen.git



repository git clone https://github.com/davidjamesca/ctypesgen.git



. Go to the previously built library build/lib/



and run the script.

 python3 ~/ctypesgen/run.py -lmal ../include/*.h -o main_wrap.py
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

After that, we check the work.

 $ python3 > import main_wrap as main > dir(main) [... 'struct_test', 'minus', 'sum'] > main.sum(main.struct_struct_test(1,2)) 3 > main.minus(main.struct_struct_test(1,2)) -1
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Well, returning to the question of already installed libraries, let's say that we want to make a wrapper for the neon library (which is already installed on the system in any convenient way), as shown in the Readme Stypesgen.

 $ ctypesgen.py -lneon /usr/local/include/neon/ne_*.h -o neon.py $ python > import neon > dir(neon) [...,'sys', 'time_t', 'union_ne_session_status_info_u', 'wstring_at']
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Finally, a link to github , how could it be without it.