Internship at the Haxe Foundation

I present to you the translation of another report from HaxeUp Sessions 2019 Linz , I believe that it well complements the previous one , as continues the theme of changes in Haxe that occurred in 2019, and also talks a bit about its future.

A little about the author of the report: Aurel Bili met with Haxe , participating in various game jams, and he continues to participate in them (for example, here is his game from the last Ludum Dare 45 ).

Aurel is currently completing studies at Imperial College London, which implies compulsory internships. The first internship he took was in a remote office, the road to which took a lot of time. Therefore, he hoped that the next practice would be possible to go through remotely.

It so happened that the Haxe Foundation for a long time could not find an employee for the position of compiler developer. Aurel decided to try his luck and sent a letter requesting remote work. He was lucky - he was accepted for a six-month internship with the opportunity to work from London.

At the device, the range of tasks that Aurel will be engaged in was agreed upon (although not everything was eventually realized).

What did he do?

Firstly, the documentation , which was in a sad state: all changes in the syntax, new features of the language and the compiler were described, sections on strings, literals and constants were supplemented.

All documentation has been translated from LaTeX to Markdown !

Secondly, the formatting of the standard library code was reduced to a single style (since different people with different styles of code design worked on it for over 10 years). Thus, in the repository of the Haxe compiler, Aurel took seventh place in the number of lines of code added :)

Thirdly, Aurel also worked on the standard library and compiler:

For example, the Map



container has a new clear()



method that removes all stored values. This was done primarily for the convenience of working with containers created as final



variables (that is, they cannot be assigned a new value, but you can modify them):

For objects of type Date



, methods for working with dates in the UTC (universal universal time) format have appeared. Work on them showed how difficult it is to implement a single API that works equally well on all 11 languages ​​/ platforms supported by Haxe.

In the old compiler, the definitions and meta tags were defined on OCaml, but now they are described in the JSON format, which should simplify their parsing by external utilities (for example, to automatically generate documentation):

You may also notice that on large projects, the compilation server begins to use a lot of memory.

To solve this problem, Simon Kraevsky and Aurel developed the hxb binary format, which is used to serialize the typed AST. Now the compilation server can load the module into memory, work with it until it is needed, and then unload it from memory to a file in hxb format and free up the occupied memory.

The hxb format specification is available in a separate repository , and its current implementation in the compiler (with serializer / deserializer) lies in a separate Haxe branch . Work on this feature has not yet been completed, and perhaps it will appear in Haxe 4.1.

The fourth and main focus of Aurel's work during the internship was the creation of a new asynchronous system API - asys.

The need for its creation is due to the fact that the existing API does not provide easy ways to perform system operations asynchronously. For example, to work with files asynchronously, you will have to create a separate thread in which the required operations will be performed, and manually control its state. In addition, the current API does not have all the functionality for working with UDP sockets, which are in standard libraries in other languages, there is no support for IPC sockets.

When creating and implementing a new API, many questions arise:

How to design an API? Maybe it’s worth taking an existing one as an example? After all, we don’t want to create everything from scratch, because it will take more time, and may also not be to the taste of the rest of the team and cause much debate.

And, as already mentioned, the actual problem for Haxe is the implementation of a single API for all supported platforms.

API Node.js. was chosen as a sample. It is well thought out, supports the necessary system functions and is well suited for creating server applications.

But at the same time, the Node.js API is a Javascript API with no strong typing. For example, functions from the fs



module for working with the file system can take as paths either strings or objects like Buffer



and even URL



. And this is not so good for Haxe.

Node.js, in turn, uses the libuv library written in C. Working with the libuv API from Haxe directly would not be so convenient: for example, in order to rename the file asynchronously, you would need to additionally create objects like uv_loop_t



(structure for managing event loop in libuv) and uv_fs_t



(structure for describing a request to the file system):

As a result, the Node.js and libuv APIs were integrated as follows (using the eval macro interpreter and rename



method as an example):

• they took the API method from Node.js, converted it to Haxe, trying to standardize the types of arguments and get rid of arguments that are redundant for Haxe. For example, path arguments ( FilePath
  
  
  
  type) are abstracts over strings:

• then created OCaml binders for this method:

• linked OCaml and C (using CFFI - C Foreign Function Interface):

• and finally wrote C-binders to call libuv C functions from OCaml:

Similarly, it was done for HashLink and Neko (for now, the asys API is implemented only for these three platforms). As can be assumed, this required a lot of work.

Aurel showed some small applications demonstrating how the asys API works.

The first example is a demonstration of asynchronously reading the contents of a file. So far, the code explicitly calls methods to initialize libuv ( hl.Uv.init()



) and start the application cycle ( hl.Uv.run()



), this is due to the fact that work on the API has not been completed (but in the future they will be added automatically):

The result of the shown code:

We see that the results of the called AsyncFileSystem.readFile()



methods are displayed in the console after the “after call” trace, which is called in the code after trying to read the contents of the files.

The second example is a demonstration of asynchronous operation with DNS and IP addresses.

In the new API, it will be much easier to determine the host name, as well as helper methods for working with IP addresses.

The third example is a simple TCP echo server, which requires only three lines of code to create:

A fourth example is a demonstration of the exchange of information between processes:

the static makeFrame()



method in this example creates separate png images:

and in the main



method, we start the ffmpeg process, into which we will transfer the frames generated in makeFrame()



:

and the output will be a video file:

And the fifth example is UDP video stream. Here, as in the previous example, the ffmpeg process is started, but this time it plays the video and outputs its data to the standard output stream. A UDP socket is also created that will broadcast data from the ffmpeg process.

And finally, we break the data received from ffmpeg into smaller “portions” and translate them to the specified port:

And as a result, we get a working video stream:

Summarizing the above, the new asys API includes:

• methods for working with the file system, including new functions that were not in the standard library (for example, for changing permissions), as well as asynchronous versions of all functions available in the old standard library
• support for asynchronous operation with TCP / UDP / IPC sockets
• methods for working with DNS (so far 2 methods: lookup
  
  
  
  and reverse
  
  
  
  )
• as well as methods for asynchronous work with processes.

Work on the asys API has not yet been completed; there are currently some problems with the garbage collector when working with the libuv library. Pull Request with the corresponding changes has not yet been incorporated into the main Haxe branch; comments on it welcome opinions regarding the names of new methods, their signatures, and documentation.

As already mentioned, support for the asys API is implemented only for HashLink, Eval, and Neko (in the form of three separate Pull Requests). Aurel has already formed a plan on how to add support for the new API for C ++ and Lua. Implementation for other platforms will require additional research.

It is possible that the asys API will become available in Haxe 4.1 (but only on some platforms).

Aurel also talked about his side project - the ammer library (which is nevertheless associated with his work at the Haxe Foundation).

Ammer's goal is to automate the creation of binders for C libraries so that they can be used in both HashLink and HXCPP (in October 2018, Lars Duse appointed a fee for solving this problem).

Why was this task relevant? The fact is that although the process of creating binders for HashLink and HXCPP is similar, for each platform you will have to write your own glue code.

Aurel did roughly the same thing when he integrated the libuv library into Haxe - for Eval, Neko and HashLink he had to write the same code, which differed only in details (function calls, differences in the work of FFI, etc.):

A similar work was required to be done on the Haxe side so that native functions could be called from it:

And the idea of ​​ammer is to take the Haxe version of the API, which is not cluttered with redundant information, and make this code somehow work for all platforms:

What ammer is now required to use external C libraries:

• create the Haxe specification for the library, which is essentially an external to the library used
• write application code
• compile the project by specifying paths to header files and C-library files
• ...
• profit

Under the hood, ammer does the following:

• matches types depending on the target platform
• automatically generates C code to call native functions
• generates a makefile that is used to create hdll, ndll files

Ammer currently supports:

• simple functions
• define'y from the header files (in the Hax-code, they can be accessed as constants)
• pointers

Support planned:

• callbacks (they are still in short supply)
• and structures (very necessary for working with the C-API)

Ammer now works with C ++, HashLink, and Eval. And Aurel is sure that he can add support for other system platforms.

To demonstrate the capabilities of ammer, Aurel showed a small application that runs the Lua interpreter:

The binders used in it are as follows:

As you can see, some methods are commented out, because they use callbacks, the support of which has not yet been realized, but Aurel hopes that he will be able to fix this soon.

So what ammer can be used for:

• embedding a Lua virtual machine
• creating applications on SDL
• automation with libuv is possible (as shown earlier, now it takes a lot of manual code to work with libuv)
• and, of course, the use of many other useful C libraries (such as OpenAL, Dear-imgui, etc.) will be greatly simplified.

Although Aurel’s internship at the Haxe Foundation has ended, he plans to continue working with Haxe, as his college education has not yet been completed and he still has to write his final work. Aurel already knows what it will be dedicated to - improving the work of the garbage collector in HashLink. Well, it will be interesting!