Hello, Habr! I present to you the translation of Steven Wolfram's post "The Wolfram Function Repository: Launching an Open Platform for Extending the Wolfram Language" .
Wolfram Language Prerequisites
Today, we are on the verge of great achievements with the Wolfram Language . Just three weeks ago, we launched the free Wolfram engine for developers to help our users integrate the Wolfram Language into their large-scale software projects. It is today that we launch the Wolfram function repository in order to provide a coordinated platform for the functions created to expand the Wolfram language, and we also open the function repository for anyone who can contribute to the development of our software product.
The Wolfram Function Repository is what became possible thanks to the unique specificity of the Wolfram Language, not only as a programming language, but also as a full-blown computational language . In traditional programming languages, the addition of significant new features usually involves the creation of entire additional libraries that may or may not work when shared. However, in Wolfram Language, so much is already built into the language itself that it is possible to significantly expand its functionality by simply adding new functions that immediately integrate into the whole structure of the entire language.
For example, now in the Wolfram function repository contains 532 new functions structured into 26 thematic categories:
Similarly to more than 6000 standard functions built into the Wolfram language, each function from the repository has a documentation page with a detailed description and examples of work:
To go to the page, copy the above object (functional BLOB), paste it into the input line and then run the function - it is already built into the Wolfram language and is supported by default, starting from version 12.0 :
It should be noted here that when processing LogoQRCode you don’t need, for example, setting up a “library for image processing” - as in Wolfram Language we already implemented a consistent and thoroughly algorithmized way of processing images that can be immediately processed with various graphic functions of the language:
I hope that with the support of a wonderful and talented community that is growing and expanding (based on the Wolfram Language) over the past few decades. In the foreseeable future, the Wolfram function repository will significantly expand the range of (possibly potentially significant, specialized in various fields of science and technology) functions available in the language. Thus, it becomes possible to use both the language content (its built-in functions) and the development principles that are implemented on the basis of the language. (It should be noted here that Wolfram Language has already more than 30 years of history of development and stable growth ).
Inside functions from the repository there may be small or rather voluminous fragments of code written in the Wolfram Language. For example, it can be calls to external APIs and services or external libraries in other languages . A unique feature of this approach is that when you switch to functionality at the user level, there will be no possible inconsistencies due to the fact that this approach is based on the agreed Wolfram Language structure - and each function will automatically work correctly - exactly as and it should be.
The shell and program structure of the Wolfram function repository are configured in such a way that everyone can contribute to the common cause in the most simple and convenient way for him - in fact, simply filling out a text file of a notepad (with the nb extension) WL . Built-in automatic functions allow you to check new functions added to the repository in order to guarantee their integration into the language. Our company relies on a wide range of users who can integrate their functions into the language, and not on the great difficulties of the new functions - and although the verification process is implemented here, we do not insist on something like a painstaking analysis of the design or strict standards for the completeness and reliability of new user functions, in contrast to more thorough checks of functions built into the core of the language that we use.
There are many compromises and details in this approach, but our goal is to optimize the Wolfram function repository both for the convenience of users and for new user functions to make a tangible contribution to the development of the language. As I grow, I have no doubt that we will have to invent new methods for processing and testing functions embedded in the repository, and this is not least for organizing a large number of functions and searching for those that users need. However, it cannot but be encouraging that the path we have chosen is a good start. I personally added several functions to the original base. Many of them are based on code that I personally developed for quite some time. And I spent only a few minutes to forward them to the repository. Now that they are in the repository, I can finally - immediately and at any time use these functions as needed, without worrying about finding files, downloading packages, etc.
Increase efficiency while lowering costs.
Even before the advent of the Internet, there were ways to exchange Wolfram Language code (our first major centralized project was MathSource , created for Mathematica in 1991 based on CD-ROM, etc.). Of course, the proposed approach based on the Wolfram function repository is a more powerful and reliable tool for implementing the above tasks.
For more than 30 years, our company has been working hard to maintain the integrity of the Wolfram language structure, and this is very important for Wolfram to become not only a programming language, but also a full-fledged computational language . And thus, the essence of the approach to the implementation of the Wolfram function repository is the use of a single approach to programming and the development of new functions that are sequentially added and fit into the framework of the language for the possibility of its development and joint evolution.
In the structure of the implementation of each function, various computational processes occur. It should be noted here that it is necessary for the user to have a clear and uniform look and clear readability. In this context, Wolfram Language's built-in functions provide more than 6,000 consecutive examples of how to properly program functions (these are our live video programming reviews that include hundreds of hours of typical programming process creation ). This approach ultimately allows you to make the Wolfram function repository capable of good work, namely the structural nature of the Wolfram Language language with its large number of additional and diverse libraries that are already built into the language. For example, if you have a function that processes images, or sparse arrays , or molecular structures , as well as geographic data or some other - the language already has a consistent symbolic representation of them, and thanks to this, your function immediately becomes compatible with other functions in the language.
Creating a repository that works really well is an interesting meta-programming task. For example, an excess of restrictions in the program will not be able to get the required unification and universality of the algorithm. Just as with an insufficient number of functional limitations, you cannot implement a sufficiently correct sequence of algorithm execution. Several previous examples of the compromise between these approaches implemented by our company have worked quite stably - these are: The Wolfram Demonstration Project , launched in 2007 and now working interactively on the Internet and containing more than 12,000 user interactive demos. The Wolfram database has more than 600 ready-made databases that can be used in the Wolfram Language language, and the Wolfram neural network store is updated with new neural networks almost every week (there are 118 of them now) and they are immediately connected via the NetModel function in the Wolfram Language language.
All of the above examples have a fundamental feature - the objects and functions collected in the project have a very high degree of structuring and distribution of processes. Of course, the granularity of the structure of what is a demonstration or a neural network or something else can vary greatly, but the fundamental structure for any current repository always remains unchanged. So what is your opinion, dear user, about creating such a repository that adds extensions to the Wolfram language? Wolfram Language is designed to be extremely flexible, so it can be expanded and modified in any way. This circumstance is extremely important for the ability to quickly create various large-scale software projects in the Wolfram Language. It should be noted here that with the flexibility of the language, the cost of projects implemented in such a language will inevitably increase. This is due to the fact that the more the user uses such a language, the more he gets the allocated functionality, but we should not forget that this approach may have negative sides in terms of the impossibility of ensuring consistent consistency of program modules.
In traditional programming languages, there is a common problem with libraries - if you, for example, use one library, then the code works correctly, but if you try to use several libraries, there is no guarantee that they will interact correctly with each other. Also, in traditional programming languages - unlike a full-fledged computing language - there is no way to guarantee the availability of consistent built-in representations for any functions or data types other than their basic structures. But, in fact, the problem is even greater than it seems at first glance: if someone is building a large-scale vertical of functionality, then without the enormous cost of centralized programming of the project that we have invested in the Wolfram language, it is impossible to achieve consistency. Therefore, it is important that all program modules always work correctly together.
Thus, the idea of the Wolfram function repository is to avoid the problem voiced above - simply adding extensions to the language in the form of relatively small elements of program code through separate functions, the development of which in the form of coordinated modules is an easier task. At the same time, there are programming features that cannot be made convenient with the help of separate functions (and our company is going to release an optimized software algorithm in the near future to help implement large-scale software packages). However, based on the functions already built into the Wolfram Language, there are many programming options that are implemented on the basis of individual functions. The idea here is that, with relatively small software efforts, it is possible to create a number of new and very useful functions that will ensure sufficient project coherence, while they will be well coordinated with each other, and, in addition, they will be able to easily and widely used in the language later on.
This approach, of course, is a compromise. With the implementation of a larger package, you can imagine a whole new world of functionality that will become extremely powerful and useful. If there is a need to get new functionalities that will fit into everything else, but at the same time you are not ready to spend huge efforts on the development of the project, this, unfortunately, can lead to a decrease in the scale of your project. The idea of the Wolfram function repository is to provide the defining part of the project with functionality, this approach will add powerful functionality, while simplifying the maintenance of good consistency of the programming project.
Help adding custom functions to the function repository
Our specialists worked hard to make their contribution to the Wolfram repository functions easy enough for users. On the desktop (already in version 12.0 ), you can simply go sequentially through the tabs of the main menu: File> New> RepositoryItem> Function Repository Item and you will get a “ Definition Notebook ” (programmatically inside the work environment. You can also use the analog function - CreateNotebook [ "FunctionResource" ] ):
There are two main actions that you will need to perform: firstly, to actually write down the code of your function and, secondly, to write documentation illustrating how your function should work.
Click the “Open Sample” button at the top to see an example of what you need to do:
In fact, you are trying to create something similar to a built-in function in the Wolfram Language. Except that it can do something much more specific than an inline function. At the same time, expectations regarding its completeness and reliability will be much lower.
You need to provide a name for your function that follows the Wolfram Language function naming conventions. In addition, you will need to develop documentation for your function, similar to the built-in language functions. I will discuss this in more detail later. Now just pay attention to the fact that in the row of buttons at the top of the definition notebook file there is a “Style Guide ” button that explains what to do and a “Tools” button that provides tools for formatting the documentation of your function.
When you are sure that everything is filled out properly and you are ready, click the “Check” button. It is perfectly normal that you have not yet figured out all the details. Therefore, the “Check” function will automatically execute and do a lot of style and consistency checks. Often, she will immediately prompt you to confirm and accept corrections (For example: “This line should end with a colon”, and she will offer to enter a colon). Sometimes she asks you to add or change something yourself. We will constantly add new features to the automatic functionality of the “Check” button, but its main goal is to guarantee that everything you send to the function repository already exactly matches as many style recommendations as possible
So, after running “Check”, you can use “Preview”. “Preview” creates a preview of the documentation page that you defined for your function. You can also create a preview for a file created on your computer or for a file located in cloud storage. If for some reason you are not satisfied with what you see in the preview, just go back and make the necessary corrections, and then click the Preview button again.
Now you are ready to place your function in the repository. The Deploy button provides you with four options:
At this step, it is important that you can send your function to the Wolfram function repository so that it is available to any user. At the same time, you can also place your function for a limited number of users. For example, you can create a function located locally on your computer so that it is available when you use this particular computer. Or you can place it in your cloud account so that it is available to you when you are connected to the cloud. You can also publicly place (deploy) a function through your cloud account. At the same time, it will not be in the central repository of Wolfram functions, but you can give someone a URL that will allow them to get your function from your account. (In the future, we will also support central repositories throughout our company).
So, let's say you want to actually transfer your function to the Wolfram function knowledge base. To do this, you click the "Send" button to the repository. So what is happening at the moment? Your application immediately enters the queue for consideration and approval by our special team of curators.
As your application goes through the approval process (which usually takes several days), you will receive messages about the status of its consideration, as well as, possibly, proposals for its further use. But as soon as your function is approved, it will be immediately published in the Wolfram function repository and will be available for any use. (And it will be displayed in news digests of new features , etc.)
What should be in storage?
It should be noted that our company has very high standards of completeness, reliability and overall quality, of the 6000+ functions that we have already built into the Wolfram language over the past 30-plus years, all meet the above requirements. The purpose of the Wolfram function repository is to use the entire structure and functionality that already exist in the Wolfram Language to add as many much lighter functions (i.e., higher performance functions) as possible.
Of course, the functions in the Wolfram function repository should be consistent with the Wolfram Language development principles - so that they can fully interact with other functions and users' expectations regarding how the function should work properly. However, the functions should not have the same completeness or reliability.
In the built-in functions of the Wolfram language, we work hard to make the program functions as general as possible. At the same time, when you are in the Wolfram function repository, there is nothing wrong with getting a function in it that simply handles some very specific, but useful case. For example, the SendMailFromNotebook function can receive files in one specific format and create mail in one specific way. PolygonalDiagram only creates diagrams with specific colors and markings, etc.
Another point related to the built-in functions is that our company makes every effort to handle all non-standard cases, to correctly handle incorrect input, and so on. In the repository of functions, it is completely normal that there is a special function in it that processes the main cases of solving the problem and ignores all the others.
The obvious fact is that it’s better to have functions that do more and do it better, but the optimization for the function repository - unlike the Wolfram built-in functions - should have more functions tied up with a lot of functions, rather than delving into the implementation processes of each specific function.
Now let's look at an example of testing functions in a repository. The expectation of consistency for such functions is naturally much lower than for the built-in language functions. This is even more relevant in cases when functions depend on external resources, such as APIs, it is important to constantly conduct sequential tests, which automatically occurs inside the verification algorithms. In the nb file, you can explicitly specify the definitions (in the "Additional Information" section) and specify as many tests, defined either by input and output lines, or by full character objects of the VerificationTest type, as you see fit. In addition, the system constantly tries to turn the examples of documentation that you provided into the verification process (sometimes it can be quite resource-intensive, for example, for a function whose result depends on random numbers or time of day).
As a result, there will be a number of implementation difficulties in the function repository. Some will be just one line of code, others may include thousands or tens of thousands of lines, probably using many helper functions. When is it worth adding a function that requires very little code to define? In principle, if a function has a good mnemonic name that users would readily understand if they saw this in the code part, then it can already be added. Otherwise, it is probably best to simply add the code to your program again each time, in those cases when you need to use it.
The main purpose of the function repository (as its name implies) is to introduce new functions into the language’s functional. If you want to add new data or new entities , use the Wolfram Data repository . But what if you want to introduce new kinds of objects for your calculations?
There are actually two ways. You might want to introduce a new type of object that will be used in new functions in the function repository. And in this case, you can always simply write down its symbolic representation and use it when entering or outputting functions in the function repository.
But what if you want to represent an object, and then determine, through existing functions in the Wolfram Language, which should work with it? To do this, Wolfram Language has always had a lightweight mechanism called UpValues . With some limitations (especially for functions that cannot evaluate their arguments ), the function repository allows you to simply represent the function and define values for it. (To increase the expectation of consistency when creating a new important design that is fully integrated everywhere in the Wolfram Language, this is usually a very important procedure that cannot be achieved only by increasing the cost of the project and this is what our company does within projects of long-term development of the language, this task is not the goal that is set in the framework of the development of the repository).
So what can be in the function code in the function repository? Everything that is built into the Wolfram Language , of course (at least if it does not pose a threat to the security and performance of the program itself, as a computing environment) as well as any function from the function repository. However, there are other functional possibilities: a function in the function repository can call the API either in Wolfram Cloud or from another source . Of course, there are some risks associated with this. Due to the fact that there are no guarantees that the API will not change, and the function in the function store will stop working. To identify such problems, there is a note on the documentation page (in the “Requirements” section) for any function that relies not only on Wolfram Language's built-in functionality. (Of course, when it comes to real data, there may be problems even with such functionality - because real world data is constantly changing, and sometimes their definitions and structure also change.)
Should all Wolfram function repository code be written in Wolfram?Certainly, the code inside the external API should not be written in the Wolfram language, which actually does not even make the language code. In fact, if you find a function in almost any external language or library, you can create a shell that allows you to use it in the Wolfram function repository. (Typically, you should use the built-in ExternalEvaluate or ExternalFunction functions in Wolfram code.)
? , Wolfram Language . , Wolfram Language , . , , . ., Wolfram Language. ( , , ).
, , , , - , , . Wolfram Language, , , , , « » , .
«»
, , Wolfram! - ( ), . , , , .
, , — — . , . ( Wolfram Enterprise , , .)
, Wolfram, ; . «» Unix — « » « », , . ., . , , , .
: « » « Wolfram ». (, ), , , , .
Wolfram, , . « », «». , , . . . , , , , , , .
Git , , , . ( ), , , , .
, «» . , , ( , ). , , . , ResourceUpdate . (« BLOB-» , .)
Wolfram , Wolfram Language . , « » (, ).
, ResourceFunction [ ... ] . , Wolfram Engine, , IDE (c , Free Wolfram Engine ).
How it works?
Wolfram , ( , Neural Net Repository , ..), Wolfram, ResourceFunction ResourceObject .
, Information :
? — . , ( ) :
, :
, BLOB- 
. , BLOB- , . , , 
. Wolfram 
.
, , «» ? -, ResourceObject ). ( IDE , ResourceObject )
LocalCache , LocalObject . CloudDeploy , — CloudPublish . ResourceRegister , ResourceFunction[ "name" ] .
Submit Function Repository, , ResourceSubmit . ( , ResourceSubmit .)
, , . , « ». , . , ( ), Wolfram.
, , , — -, , , . , « », , , .
: « », , , MyFunction , Wolfram Language, ResourceFunction[ "MyFunction" ] – , .
: , , , . , , . (, , , .)
. . (, Method ImageSize ) , , . . , , , ( - , ). . , ( , OptionValue OptionsPattern ) , , .
, , , , — «» «».
, , , ( ) , , ResourceFunction[ "name" ] , ( ) . ( – )
. , - ?
Wolfram, , () , , , , .
. , , Wolfram Language, , . , , , , ResourceFunction[ "MyFavoriteFunction" ] .
, , , , - . , — , Wolfram. ( .) , - , , .
Wolfram , , , , , Wolfram Language. 30 Wolfram Language, (?) . , . , , «» , , , - .
( , - . , , , , , .)
Wolfram , — , ( . .). Wolfram Language , «» . . ? , .
For built-in functions in the Wolfram Language, there is a so-called detection level provided by a network of “help pages” that provide organized lists of functions related to specific areas. It is always difficult to properly balance the manual pages, and as Wolfram grows, the manual pages often need to be completely reorganized. It’s quite simple to put functions from the repository into broad categories and even break down these categories sequentially, but it’s much more valuable to have properly organized language manual pages. It is not yet clear how best to create them for the entire knowledge base of functions. For example, CreateResourceObjectGallery In the repository of functions, everyone can place a web page containing their "choices" from the repository:
The Wolfram function repository is configured as a permanent repository of functions, where any function in it will always work. Of course, new versions of functions may appear, and we expect that some functions will certainly become obsolete over time. Functions will work if they are used in programs, but their documentation pages will link to new, more advanced functions.
The Wolfram Feature Repository is designed to quickly get new features and explore new uses for the Wolfram language. With a great deal of optimism, we hope that part of what has been investigated in the function repository will ultimately make sense to become embedded parts of the Wolfram Language main language. Over the past decade, we had a similar set of features that were originally introduced in Wolfram | Alpha And one of the lessons learned from this experience is that it takes a lot of work to achieve the quality and consistency standards that we focus on in everything that is built into Wolfram, which is often more difficult than the initial effort to introducing ideas. But even in this case, a function in the knowledge base of functions can serve as a very useful proof of the concept of a future function, which in the end can be built into the Wolfram language.
The most important thing here is that the function in the function repository is what is available to every user for use right now. The built-in language function may be much better and more productive, but the function repository will allow users to immediately access all the new functions. And, most importantly, this concept allows everyone to add any of their new features.
Earlier in the history of the Wolfram language, this idea would not have worked as well as it is now, but at this stage so much effort has already been put into the language, as well as such a deep understanding of the principles of language design, that now it seems very possible for a large community of users to add functions that will Maintain project coherence to make them useful to a wide range of users.
The Wolfram Language user community has an incredible spirit of talent (?). (Of course, this community includes many leading R&D experts in a wide range of fields.) I hope that the Wolfram Feature Repository provides an effective platform for unleashing and disseminating this spirit of talent. Only together we can create something that will significantly expand the area to which the Wolfram computational paradigm can be applied.
For more than 30 years, we have come a long way in the Wolfram language. Now together, let's go even further. I urge all respected Wolfram users around the world to use a functional repository as well as a new software project like the Free Wolfram Engine for developers as a platform for this.