Taming Gorynych, or Decompiling eBPF in Ghidra

Author of the article https://github.com/Nalen98

Good afternoon!

The topic of my research as part of the Summer of Hack 2019 summer internship at Digital Security was “Decompiling eBPF in Ghidra.” It was necessary to develop an eBPF bytecode translation system in PCode Ghidra in Sleigh for the ability to disassemble and decompile eBPF programs. The result of the study is a developed extension for Ghidra that adds support for the eBPF processor. The study, like that of other interns, can rightfully be considered “first-time”, since earlier it was not possible to decompile eBPF in other reverse engineering tools.

Background

This topic went to me in a great irony of fate, because I was not familiar with eBPF before, and Ghidr was not used before, because there was a certain dogma that "IDA Pro is better." As it turned out, this is not entirely true.

Acquaintance with Ghidra turned out to be very rapid, since its developers drew up very competent and accessible documentation. Also, I had to master the Sleigh processor specification language, on which the development was carried out. The developers did their best and created very detailed documentation for both the tool itself and for Sleigh , for which many thanks to them.

On the other side of the barricade was an extended Berkeley Packet Filter. eBPF is a virtual machine in the Linux kernel that allows you to load arbitrary user code that can be used to trace processes and filter packets in kernel space. The architecture is a RISC register machine with 11 64-bit registers, a software counter and a 512-byte stack. There are a number of limitations to eBPF:

• cycles are prohibited;
• access to memory is possible only through the stack (there will be a separate story about it);
• kernel functions are available only through special wrapper functions (eBPF-helpers).

The structure of eBPF technology. Image source: http://www.brendangregg.com/ebpf.html .

Basically, this technology is used for network tasks - debugging, packet filtering, and so on at the kernel level. EBPF support has been added since version 3.15 of the kernel; quite a few reports were devoted to this technology at Linux plumbers conference 2019. But at eBPF, unlike Ghidra, the documentation is incomplete and does not contain much. Therefore, clarifications and missing information had to be searched on the Internet. It took quite a while to find the answers, and all that remains is to hope that the technology will be finalized and normal documentation will be created.

Bad documentation

In order to develop a specification for Sleigh, you first need to understand how the architecture of the target processor works. And here we turn to the official documentation .

It contains a number of flaws:

• The structure of the eBPF instructions is not fully described.

  
  
  
  
  
  
  

  Most specifications, such as Intel x86, usually indicate what each instruction bit goes to, which block it belongs to. Unfortunately, in the eBPF specification, these details are either scattered throughout the document or absent altogether, as a result of which we have to draw the missing grains from the implementation details in the Linux kernel.

  
  
  
  
  
  
  

  For example, in the instruction structure op:8, dst_reg:4, src_reg:4, off:16, imm:32
  
  
  
  it is not said that offset (off) and immediate (imm) are signed
  
  
  
  , and this is extremely important, because it affects to work from arithmetic instructions to jumps. The source code for the Linux kernel helped.

  
  
  
  
  
  
  

• There is no complete picture of all possible mnemonics of architecture.

  
  
  
  
  
  
  

  Some documentation not only specifies all instructions, their operands, but also their semantics in C, application cases, operand features, and so on. The eBPF documentation contains instruction classes, but this is not enough for the developer. Let's consider them in more detail.

  
  
  
  
  
  
  

  All instructions of eBPF are 64-bit, except for LDDW
  
  
  
  (Load double word), it has a size of 128 bits, it concatenates two imm with 32 bits each. eBPF instructions have the following structure.
  
  
  
  eBPF instruction encoding

  
  
  
  
  
  
  

  The structure of the OPAQUE
  
  
  
  field depends on the class of instructions (ALU / JMP, Load / Store).

  
  
  
  
  
  
  

  For example, the ALU
  
  
  
  instruction class:
  
  
  
  ALU instructions encoding

  
  
  
  
  
  
  

  and the JMP
  
  
  
  class have their own field structure:
  
  
  
  Branch instructions encoding

  
  
  
  
  
  
  

  For Load / Store instructions, the structure is different:
  
  
  
  Load / Store instructions encoding

  
  
  
  
  
  
  

  Unofficial eBPF documentation helped to sort this out .

  
  
  
  
  
  
  

• There is no information about call-helpers, on which most of the logic of eBPF programs for the Linux kernel is built.

  
  
  
  
  
  
  

  And this is extremely strange, since helpers are the most important thing in eBPF programs, they just perform the tasks that the technology is focused on.

  
  
  
  
  
  
  

EBPF Interoperability with Nuclear Functions

The program pulls these functions from the kernel, and they just work with processes, manipulate network packets, work with eBPF maps, access sockets, interact with userspace. Despite the fact that the functions are still nuclear, in the official documentation it would be worth writing in more detail about them. Full details are found in the Linux source .

• Not a word about tail calls.

EBPF tail calls. Image source: https://cilium.readthedocs.io/en/latest/bpf/#tail-calls .

Tail calls are a mechanism that allows one eBPF program to call another without returning to the previous one, that is, jumping between different eBPF programs. They are not implemented in the developed extension; detailed information can be found in the Cilium documentation .

Poor documentation and a number of architectural features of eBPF were the main "splinters" in development, as they created other problems. Fortunately, most of them were resolved successfully.

About the development environment

Not all developers know that for creating and editing Sleigh code and generally all extension / plugin files for Ghidra there is a rather convenient tool - Eclipse IDE with support for GhidraDev and GhidraSleighEditor plugins . When creating the extension, it will be immediately framed as a working draft, there is a rather convenient highlight for the Sleigh-code, as well as a checker of the main errors in the language syntax.

In Eclipse, you can run Ghidra (already with the extension turned on), debug, which is extremely convenient. But perhaps the coolest opportunity is to support the "Ghidra Headless" mode, you do not need to restart Ghidr from the GUI 100500 times to find an error in the code, all processes are carried out in the background.

Notepad can be closed! And you can download Eclipse from the official site . To install the plugin, in Ecplise, select Help → Install New Software ... , click Add and select the plugin zip archive.

Extension development

For the extension, processor specification files were developed, a loader that inherits from the main ELF loader and expands its capabilities in terms of recognizing eBPF programs, a relocation processor for implementing eBPF Maps in the Ghidra disassembler and decompiler, as well as an analyzer for determining eBPF helper signatures.

Extension files as a project in the Eclipse IDE

Now about the main files:

.cspec



- it indicates which data types are used, how much memory is allocated to them in eBPF, the stack size is set, the “stackpointer” label is set to register R10



, and the call agreement is signed. The agreement (like the rest) was implemented according to the documentation:

Therefore, eBPF calling convention is defined as:

• R0 - return value from in-kernel function, and exit value for eBPF program
• R1 - R5 - arguments from eBPF program to in-kernel function
• R6 - R9 - callee saved registers that in-kernel function will preserve
• R10 - read-only frame pointer to access stack

 <?xml version="1.0" encoding="UTF-8"?> <compiler_spec> <data_organization> <absolute_max_alignment value="0" /> <machine_alignment value="2" /> <default_alignment value="1" /> <default_pointer_alignment value="4" /> <pointer_size value="4" /> <wchar_size value="4" /> <short_size value="2" /> <integer_size value="4" /> <long_size value="4" /> <long_long_size value="8" /> <float_size value="4" /> <double_size value="8" /> <long_double_size value="8" /> <size_alignment_map> <entry size="1" alignment="1" /> <entry size="2" alignment="2" /> <entry size="4" alignment="4" /> <entry size="8" alignment="8" /> </size_alignment_map> </data_organization> <global> <range space="ram"/> <range space="syscall"/> </global> <stackpointer register="R10" space="ram"/> <default_proto> <prototype name="__fastcall" extrapop="0" stackshift="0"> <input> <pentry minsize="1" maxsize="8"> <register name="R1"/> </pentry> <pentry minsize="1" maxsize="8"> <register name="R2"/> </pentry> <pentry minsize="1" maxsize="8"> <register name="R3"/> </pentry> <pentry minsize="1" maxsize="8"> <register name="R4"/> </pentry> <pentry minsize="1" maxsize="8"> <register name="R5"/> </pentry> </input> <output killedbycall="true"> <pentry minsize="1" maxsize="8"> <register name="R0"/> </pentry> </output> <unaffected> <varnode space="ram" offset="8" size="8"/> <register name="R6"/> <register name="R7"/> <register name="R8"/> <register name="R9"/> <register name="R10"/> </unaffected> </prototype> </default_proto> </compiler_spec>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Before continuing to describe development files, I will .cspec



on a small line of the .cspec



file.

 <stackpointer register="R10" space="ram"/>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

It is the main source of evil when decompiling eBPF in Ghidra, and it began an exciting journey into the eBPF stack, which has a number of unpleasant moments, and which brought the most pain to the development.

All we need is ... Stack

Let's look at the official kernel documentation :

Q: Can BPF programs access instruction pointer or return address?



A: NO.



Q: Can BPF programs access stack pointer?



A: NO. Only frame pointer (register R10) is accessible. From compiler point of view it's necessary to have stack pointer. For example, LLVM defines register R11 as stack pointer in its BPF backend, but it makes sure that generated code never uses it.

The processor has neither an instruction pointer (IP) nor a stack pointer (SP), and the latter is extremely important for Ghidra, and the quality of decompilation depends on it. In the cspec



file, you need to specify which register is the stackpointer (as demonstrated above). R10



is the only eBPF register that allows accessing the program stack, it is framepointer, it is static and always zero. Hanging the “stackpointer” label on R10



in the cspec



file is fundamentally wrong, but there are no other options, since then Ghidra will not work with the program stack. Accordingly, the original SP is absent, and nothing replaces it in the eBPF architecture.

Several issues arise from this:

1. The "Stack Depth" field in Ghidra will be guaranteed to be zero, since we simply have to designate R10
   
   
   
   stacker in these architectural conditions, and in essence it is always zero, which was argued earlier. "Stack Depth" will reflect the register with the label "stackpointer".

   
   
   
   
   
   
   

   And you have to put up with it, these are the features of architecture.

   
   
   
   
   
   
   

2. Instructions that operate on R10
   
   
   
   (that is, working with the stack) are often not decompiled. Ghidra generally does not decompile what it considers dead code (that is, snippets that never execute). And since R10
   
   
   
   immutable, many store / load instructions are recognized by Ghidr as deadcode and disappear from the decompiler.

   
   
   
   
   
   
   

   Fortunately, this problem was solved by writing a custom analyzer, as well as declaring an additional address space with eBPF helpers in a pspec
   
   
   
   file, which was prompted by one of the Ghidra developers in the Issue project .

   
   
   
   
   
   
   

Extension development (continued)

.ldefs



describes the features of the processor, defines specification files.

 <?xml version="1.0" encoding="UTF-8"?> <language_definitions> <language processor="eBPF" endian="little" size="64" variant="default" version="1.0" slafile="eBPF.sla" processorspec="eBPF.pspec" id="eBPF:LE:64:default"> <description>eBPF processor 64-bit little-endian</description> <compiler name="default" spec="eBPF.cspec" id="default"/> <external_name tool="DWARF.register.mapping.file" name="eBPF.dwarf"/> </language> </language_definitions>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

The .opinion



file .opinion



loader to the processor.

 <opinions> <constraint loader="Executable and Linking Format (ELF)" compilerSpecID="default"> <constraint primary="247" processor="eBPF" endian="little" size="64" /> </constraint> </opinions>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

A program counter is declared in .pspec, but with eBPF it is implicit and is not used in the specification in any way, therefore it is only for pro forma purposes. By the way, the PC



at eBPF is arithmetic, not address (it indicates an instruction, not a specific byte of the program), keep this in mind when jumping.

The file also contains an additional address space for eBPF helpers, here they are declared as characters.

 <?xml version="1.0" encoding="UTF-8"?> <processor_spec> <programcounter register="PC"/> <default_symbols> <symbol name="bpf_unspec" address="syscall:0x0"/> <symbol name="bpf_map_lookup_elem" address="syscall:0x1"/> <symbol name="bpf_map_update_elem" address="syscall:0x2"/> <symbol name="bpf_map_delete_elem" address="syscall:0x3"/> <symbol name="bpf_probe_read" address="syscall:0x4"/> <symbol name="bpf_ktime_get_ns" address="syscall:0x5"/> <symbol name="bpf_trace_printk" address="syscall:0x6"/> <symbol name="bpf_get_prandom_u32" address="syscall:0x7"/> <symbol name="bpf_get_smp_processor_id" address="syscall:0x8"/> <symbol name="bpf_skb_store_bytes" address="syscall:0x9"/> <symbol name="bpf_l3_csum_replace" address="syscall:0xa"/> <symbol name="bpf_l4_csum_replace" address="syscall:0xb"/> <symbol name="bpf_tail_call" address="syscall:0xc"/> <symbol name="bpf_clone_redirect" address="syscall:0xd"/> <symbol name="bpf_get_current_pid_tgid" address="syscall:0xe"/> <symbol name="bpf_get_current_uid_gid" address="syscall:0xf"/> <symbol name="bpf_get_current_comm" address="syscall:0x10"/> <symbol name="bpf_get_cgroup_classid" address="syscall:0x11"/> <symbol name="bpf_skb_vlan_push" address="syscall:0x12"/> <symbol name="bpf_skb_vlan_pop" address="syscall:0x13"/> <symbol name="bpf_skb_get_tunnel_key" address="syscall:0x14"/> <symbol name="bpf_skb_set_tunnel_key" address="syscall:0x15"/> <symbol name="bpf_perf_event_read" address="syscall:0x16"/> <symbol name="bpf_redirect" address="syscall:0x17"/> <symbol name="bpf_get_route_realm" address="syscall:0x18"/> <symbol name="bpf_perf_event_output" address="syscall:0x19"/> <symbol name="bpf_skb_load_bytes" address="syscall:0x1a"/> <symbol name="bpf_get_stackid" address="syscall:0x1b"/> <symbol name="bpf_csum_diff" address="syscall:0x1c"/> <symbol name="bpf_skb_get_tunnel_opt" address="syscall:0x1d"/> <symbol name="bpf_skb_set_tunnel_opt" address="syscall:0x1e"/> <symbol name="bpf_skb_change_proto" address="syscall:0x1f"/> <symbol name="bpf_skb_change_type" address="syscall:0x20"/> <symbol name="bpf_skb_under_cgroup" address="syscall:0x21"/> <symbol name="bpf_get_hash_recalc" address="syscall:0x22"/> <symbol name="bpf_get_current_task" address="syscall:0x23"/> <symbol name="bpf_probe_write_user" address="syscall:0x24"/> </default_symbols> <default_memory_blocks> <memory_block name="eBPFHelper_functions" start_address="syscall:0" length="0x200" initialized="true"/> </default_memory_blocks> </processor_spec>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

.sinc



file is the most voluminous extension file, all registers, the structure of the eBPF instruction, tokens, mnemonics and semantics of instructions in Sleigh are defined here.

 define space ram type=ram_space size=8 default; define space register type=register_space size=4; define space syscall type=ram_space size=2; define register offset=0 size=8 [ R0 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 PC ]; define token instr(64) imm=(32, 63) signed off=(16, 31) signed src=(12, 15) dst=(8, 11) op_alu_jmp_opcode=(4, 7) op_alu_jmp_source=(3, 3) op_ld_st_mode=(5, 7) op_ld_st_size=(3, 4) op_insn_class=(0, 2) ; #We'll need this token to operate with LDDW instruction, which has 64 bit imm value define token immtoken(64) imm2=(32, 63) ; #To operate with registers attach variables [ src dst ] [ R0 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 _ _ _ _ _ ]; … :ADD dst, src is src & dst & op_alu_jmp_opcode=0x0 & op_alu_jmp_source=1 & op_insn_class=0x7 { dst=dst + src; } :ADD dst, imm is imm & dst & op_alu_jmp_opcode=0x0 & op_alu_jmp_source=0 & op_insn_class=0x7 { dst=dst + imm; } …
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

The eBPF loader extends the basic capabilities of the ELF loader so that it can recognize that the program you loaded into Ghidra has an eBPF processor. For him, a BPF constant is allocated in ElfConstants



Ghidra, and the loader determines the eBPF processor from it.

 package ghidra.app.util.bin.format.elf.extend; import ghidra.app.util.bin.format.elf.*; import ghidra.program.model.lang.*; import ghidra.util.exception.*; import ghidra.util.task.TaskMonitor; public class eBPF_ElfExtension extends ElfExtension { @Override public boolean canHandle(ElfHeader elf) { return elf.e_machine() == ElfConstants.EM_BPF && elf.is64Bit(); } @Override public boolean canHandle(ElfLoadHelper elfLoadHelper) { Language language = elfLoadHelper.getProgram().getLanguage(); return canHandle(elfLoadHelper.getElfHeader()) && "eBPF".equals(language.getProcessor().toString()) && language.getLanguageDescription().getSize() == 64; } @Override public String getDataTypeSuffix() { return "eBPF"; } @Override public void processGotPlt(ElfLoadHelper elfLoadHelper, TaskMonitor monitor) throws CancelledException { if (!canHandle(elfLoadHelper)) { return; } super.processGotPlt(elfLoadHelper, monitor); } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

The relocation handler is required to implement eBPF maps in the disassembler and decompiler. Interaction with them is carried out through a number of helpers, functions use a file descriptor to indicate maps. Based on the relocation table, it can be seen that the loader patches the LDDW instruction, which generates Rn



for these helpers (for example, bpf_map_lookup_elem(…)



).

Therefore, the handler parses the program relocation table, finds the relocation addresses (instructions), and also collects string information about the map name. Further, referring to the symbol table, it calculates the real addresses of these maps and patches the instructions.

 public class eBPF_ElfRelocationHandler extends ElfRelocationHandler { @Override public boolean canRelocate(ElfHeader elf) { return elf.e_machine() == ElfConstants.EM_BPF; } @Override public void relocate(ElfRelocationContext elfRelocationContext, ElfRelocation relocation, Address relocationAddress) throws MemoryAccessException, NotFoundException { ElfHeader elf = elfRelocationContext.getElfHeader(); if (elf.e_machine() != ElfConstants.EM_BPF) { return; } Program program = elfRelocationContext.getProgram(); Memory memory = program.getMemory(); int type = relocation.getType(); int symbolIndex = relocation.getSymbolIndex(); long value; boolean appliedSymbol = true; //Relocations with maps always have type 0x1. Since eBPF hasn't names of constants (types) of relocations, it was decided to use magic //number 1. if (type == 1) { try { int SymbolIndex= relocation.getSymbolIndex(); ElfSymbol Symbol = elfRelocationContext.getSymbol(SymbolIndex); String map = Symbol.getNameAsString(); SymbolTable table = program.getSymbolTable(); Address mapAddr = table.getSymbol(map).getAddress(); String sec_name = elfRelocationContext.relocationTable.getSectionToBeRelocated().getNameAsString(); if (sec_name.toString().contains("debug")) { return; } value = mapAddr.getAddressableWordOffset(); Byte dst = memory.getByte(relocationAddress.add(0x1)); memory.setLong(relocationAddress.add(0x4), value); memory.setByte(relocationAddress.add(0x1), (byte) (dst + 0x10)); } catch(NullPointerException e) {} } if (appliedSymbol && symbolIndex == 0) { markAsWarning(program, relocationAddress, Long.toString(type), "applied relocation with symbol-index of 0", elfRelocationContext.getLog()); } } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

The result of disassembling and decompiling eBPF

And in the end, we get the eBPF disassembler and decompiler! Use for health!

Extension on GitHub: eBPF for Ghidra .

Releases here: here .

PS

Many thanks to Digital Security for an interesting internship, especially to the mentors from the research department (Alexander and Nikolai). I bow to you!