JavaScript Extensible Extensibility Mechanisms
Hello colleagues!
We remind you that not so long ago we published the 3rd edition of the legendary book “Expressive JavaScript ” (Eloquent JavaScript) - it was printed in Russian for the first time, although high-quality translations of previous editions were found on the Internet.
However, neither JavaScript nor the research work of Mr. Haverbeke, of course, does not stand still. Continuing the topic of expressive JavaScript, we offer a translation of the article on the design of extensions (using the development of a text editor as an example), published on the author’s blog in late August 2019
Nowadays, it has become fashionable to structure large systems in the form of many separate packages. The driving idea underlying this approach is that it’s better not to restrict people to a specific feature (proposed by you) by implementing a feature, but to provide this feature as a separate package that a person can download along with the basic system package.
To do this, in general terms, you will need ...
This approach involves certain costs, which boil down to additional complexity. So that users can get started, you can provide them with a wrapper package, which is all-inclusive, but at some point they will probably have to remove this wrapper and tackle the installation and configuration of auxiliary packages on their own, and this turns out to be more difficult than including New feature delivered in the monolithic library.
In this article, I will try to explore various ways of designing extensibility mechanisms that involve “extensibility on a grand scale” and immediately lay new points for future expansion.
Extensibility
What do we need from an extensible system? First, of course, you need the ability to build new behaviors over external code.
However, this is hardly enough. Let me digress, talk about one stupid problem that I once had a chance to encounter. I am developing a text editor. In one earlier version of the code editor, the client could specify the appearance of a particular line. It was excellent - the user could selectively select this or that line in this way.
In addition, if you try to initiate a line design from two mutually independent code fragments, then they begin to step on each other's heels. The second extension, which applies to a particular line, overwrites the changes made through the first. Or, if at some later point in time we try to remove the design changes that we made with the help of the first code, then as a result we will overwrite the design made from the second code fragment.
The solution was to allow code to be added (and removed ) rather than installed, so that two extensions could interact with the same line without interrupting each other's work.
In a more general formulation, it is necessary to make sure that extensions can be combined, even if they are “completely unaware” of each other — and so that no conflicts arise during their interactions.
To do this, each extension must be exposed to any number of agents simultaneously. How exactly each effect will be processed differs depending on the specific case. Here are some strategies you might find useful:
In many such situations, order is important. Here I mean compliance with the order in which the effects are applied, this sequence must be controlled and predictable.
This is one of those situations where imperative extension systems are usually not very good, the operation of which depends on side effects. For example, the
operation of the
DOM model requires that event handlers be called in the order in which they are registered. This is normal if all calls are controlled by a single system, or if the order of the calls is not so important. However, if you have many software components that add handlers independently of each other, it can be very difficult to predict which ones will be called in the first place.
Simple approach
Let me give you a concrete example: I first applied such a modular strategy while developing ProseMirror, a system for editing rich text. The core of this system, in itself, is essentially useless: it relies on additional packages to describe the structure of documents, to bind keys, to maintain a history of cancellations. Although using this system is really a little difficult, it has found application in programs where you need to configure things that are not supported in classic editors.
The ProseMirror extension mechanism is relatively straightforward. When creating an editor, a single array of connected objects is specified in the client code. Each of these plug-in objects can affect various aspects of the editor and do things such as adding bits of status data or handling interface events.
All these aspects were designed to work with an ordered array of configuration values, using one of the strategies described above. For example, when you need to specify a lot of dictionaries with values, the priority of following extension instances for key binding depends on the order in which you specify these instances. The first extension for key binding, knowing what to do with this keystroke, gets it processed.
Usually, such a mechanism turns out to be quite powerful, and they can turn it to their advantage. But sooner or later, the extension system reaches such complexity that it becomes inconvenient to use it.
CodeMirror version 6 is a rewritten version of the code editor of the same name. In this project, I try to develop a modular approach. To do this, I need a more expressive extension system. Let's discuss some of the challenges that we had to deal with when designing such a system.
Ordering
It’s easy to design a system that gives you complete control over the ordering of extensions. However, it is much more difficult to design a system with which it will be pleasant to work with and which, at the same time, will allow you to combine the code of various extensions without numerous interventions from the “now watch your hands” category.
When it comes to ordering, it happens, I really want to resort to work with priority values. As an example, the CSS
property indicates the number of the position occupied by this element in the depth of the stack.
As you can see on the example of ridiculously large
values that are sometimes found in style sheets, this way of indicating priority is problematic. The module itself does not know what priority values other modules have. Options are just dots in the middle of an undifferentiated numerical range. You can set a huge (or deeply negative) value to try to reach the far edges of this spectrum, but the rest of the work comes down to fortune telling.
The situation can be improved a little if you define a limited set of clearly defined priority categories, so that extensions can characterize the general “level” of their priority. In addition, you will need a certain way to break the links within the categories.
Grouping and Deduplication
As I mentioned above, as soon as you start to rely seriously on extensions, a situation may arise when some extensions will use others when working. If you manage dependencies manually, then this approach does not scale well; therefore, it would be nice if you could pull up a group of extensions at the same time.
However, this approach not only further aggravates the priority problem, but also introduces another problem: many other extensions may depend on a particular extension, and if the extensions are presented as values, it may well be that the same extension will be loaded multiple times . For some types of extensions, such as event handlers, this is normal. In other cases, such as with a cancellation history and a tooltip library, this approach will be wasteful and may even break everything.
So, if we allow the layout of extensions, this introduces into our system a certain additional complexity associated with dependency management. You must be able to recognize such extensions that should not be duplicated, and download them exactly one at a time.
However, since in most cases extensions can be configured, and therefore not all instances of the same extension will be exactly the same, we cannot just take one instance and work with it. We will have to consider some meaningful merger of such instances (or report an error if the merge of interest to us is impossible).
Project
Here I will describe in general terms what we are doing in CodeMirror 6. This is just a sketch, not a Failed Solution. It is possible that this system will develop further when the library stabilizes.
The main primitive used in this approach is called behavior. Behaviors are just those features that you can build on with extensions, specifying values for them. An example is the behavior of a state field where, with the help of extensions, you can add new fields, providing a description of the field. Another example is the behavior of event handlers in a browser; in this case, with the help of extensions, we can add our own handlers.
From the point of view of the consumer of behaviors, the behaviors themselves, configured in a certain way in a particular instance of the editor, give an ordered sequence of values, and those values that come before have a higher priority. Each behavior has a type, and the values provided for it must match that type.
The behavior is represented as a value used both to declare an instance of the behavior and to access the values that the behavior has. There are a number of built-in behaviors in the library, but external code can define its own behaviors. For example, in the extension defining the interval between line numbers, behavior can be defined that allows another code to add additional markers in this interval.
An extension is a value that can be used when configuring the editor. An array of such values is passed during initialization. Each extension is allowed in zero or more behaviors.
Such a simple extension can be considered an instance of behavior. If we specify a value for the behavior, then the code returns us the value of the extension that generates this behavior.
A sequence of extensions can also be grouped into a single extension. For example, in the editor’s configuration for working with a specific programming language, you can pull up several other extensions - for example, grammar for parsing and highlighting text, information about the necessary indentation, an autocompletion source that will correctly display prompts for completing lines in this language. Thus, you can make a single language extension in which we simply collect all these corresponding extensions and group them together, resulting in a single value.
When creating a simple version of such a system, we could stop at this by simply aligning all the nested extensions in one array of behavior extensions. Then they could be grouped by type of behavior and then build ordered sequences of values of behavior.
However, it remains to deal with deduplication and provide better control over ordering.
The values of extensions related to the third type, unique extensions, just help to achieve deduplication. Extensions that should not be instantiated twice in the same editor are of this kind. To determine such an extension, you need to specify the spec-type , that is, the type of configuration value expected by the extension constructor, and also specify an instantiation function that takes an array of such specified values and returns the extension.
Unique extensions complicate the process of resolving a set of extensions into a set of behaviors. If there are unique extensions in the aligned set of extensions, then the resolving mechanism should select the type of unique extension, collect all its instances and call the corresponding instantiation function along with the specifications, and then replace them all with the result (in a single copy).
(There’s another catch: they must resolve in the correct order. If you first allow the unique extension X, but then get another X as a result of the resolution, then this will be wrong, since all instances of X must be put together. Since the instantiation function expansion is clean, the system copes with this situation by trial and error, restarting the process and recording information about what it was possible to study, being in this situation.)
Finally, you need to resolve the issue with the rules of the sequence. The basic approach remains the same: maintain the order in which extensions were proposed. Compound extensions align in the same order at the point where they occur. The result of resolving a unique extension is inserted when it is first turned on.
However, extensions can relate some of their sub extensions to categories that have a different priority. The system provides for four such categories: fallback (takes effect after other things happen), default (default), extend (higher priority than the bulk) and override (probably should go first). In practice, sorting is done first by category, and then by starting position.
So, a key binding extension, which has a low priority, and an event handler with a normal priority, it’s fashionable to get a compound extension based on the result of a key binding extension (not requiring knowing what behavior it consists of) with a fallback priority level and from an instance with the behavior of the event handler.
This approach, which allows you to combine extensions without thinking about what they do “inside,” seems to be a great achievement. The extensions we modeled earlier in this article include two parsing systems that exhibit the same behavior at the syntax level, syntax highlighting service, smart indentation service, cancellation history, line-spacing service, auto-closing brackets, key binding and multiple selection - all works good.
There are several new concepts that the user must learn in order to use this system. In addition, working with such a system is indeed a bit more complicated than with traditional imperative systems adopted in the JavaScript community (we call a method to add / remove an effect). However, if the extensions are properly arranged, then the benefits of this outweigh the associated costs.
We remind you that not so long ago we published the 3rd edition of the legendary book “Expressive JavaScript ” (Eloquent JavaScript) - it was printed in Russian for the first time, although high-quality translations of previous editions were found on the Internet.
However, neither JavaScript nor the research work of Mr. Haverbeke, of course, does not stand still. Continuing the topic of expressive JavaScript, we offer a translation of the article on the design of extensions (using the development of a text editor as an example), published on the author’s blog in late August 2019
Nowadays, it has become fashionable to structure large systems in the form of many separate packages. The driving idea underlying this approach is that it’s better not to restrict people to a specific feature (proposed by you) by implementing a feature, but to provide this feature as a separate package that a person can download along with the basic system package.
To do this, in general terms, you will need ...
- The ability to not even download features you don’t need, which is especially useful when working with client-side systems.
- The ability to replace with another implementation the functionality that does not meet your needs. In this way, pressure is also reduced on the nuclear modules, which otherwise would have to cover all sorts of practical cases.
- Checking kernel interfaces in real conditions - by implementing basic features on top of the client-facing interface; you will have to make the interface powerful enough so that it at least cope with the support of these features - and make sure that functionally similar elements can be assembled from third-party code.
- Isolation between system components. Project participants will need to look for the specific package that interests them. Packages can be versioned, deprecated or replaced by others, and all this will not affect the kernel.
This approach involves certain costs, which boil down to additional complexity. So that users can get started, you can provide them with a wrapper package, which is all-inclusive, but at some point they will probably have to remove this wrapper and tackle the installation and configuration of auxiliary packages on their own, and this turns out to be more difficult than including New feature delivered in the monolithic library.
In this article, I will try to explore various ways of designing extensibility mechanisms that involve “extensibility on a grand scale” and immediately lay new points for future expansion.
Extensibility
What do we need from an extensible system? First, of course, you need the ability to build new behaviors over external code.
However, this is hardly enough. Let me digress, talk about one stupid problem that I once had a chance to encounter. I am developing a text editor. In one earlier version of the code editor, the client could specify the appearance of a particular line. It was excellent - the user could selectively select this or that line in this way.
In addition, if you try to initiate a line design from two mutually independent code fragments, then they begin to step on each other's heels. The second extension, which applies to a particular line, overwrites the changes made through the first. Or, if at some later point in time we try to remove the design changes that we made with the help of the first code, then as a result we will overwrite the design made from the second code fragment.
The solution was to allow code to be added (and removed ) rather than installed, so that two extensions could interact with the same line without interrupting each other's work.
In a more general formulation, it is necessary to make sure that extensions can be combined, even if they are “completely unaware” of each other — and so that no conflicts arise during their interactions.
To do this, each extension must be exposed to any number of agents simultaneously. How exactly each effect will be processed differs depending on the specific case. Here are some strategies you might find useful:
- All changes take effect. For example, if we add a CSS class to an element, or display a widget at a given position in the text, these two things can be done right away. Of course, some sort of ordering will be required: widgets should be displayed in a predictable and well-defined sequence.
- Changes line up in the pipeline. An example of this approach is a handler that can filter changes made to a document before they take effect. Each handler is fed a change made by the previous handler, and the subsequent handler can continue such modifications. The ordering here is not crucial, but can be significant.
- The first-come-first-served approach. This approach is applicable, for example, with event handlers. Each handler gets the opportunity to tinker with the event until one of the handlers announces that everything is already done, and then the next handler in turn will not be disturbed.
- Sometimes it is necessary to select only one value, for example, to determine the value of a specific configuration parameter. Here it would be appropriate to use some kind of operator (for example, logical or, logical and, minimum, maximum) in order to reduce the potential input to a single value. For example, the editor can switch to read-only mode if at least one extension requires it, or the maximum length of the document can be the minimum of all the values provided for this option.
In many such situations, order is important. Here I mean compliance with the order in which the effects are applied, this sequence must be controlled and predictable.
This is one of those situations where imperative extension systems are usually not very good, the operation of which depends on side effects. For example, the
addEventListener
operation of the
addEventListener
DOM model requires that event handlers be called in the order in which they are registered. This is normal if all calls are controlled by a single system, or if the order of the calls is not so important. However, if you have many software components that add handlers independently of each other, it can be very difficult to predict which ones will be called in the first place.
Simple approach
Let me give you a concrete example: I first applied such a modular strategy while developing ProseMirror, a system for editing rich text. The core of this system, in itself, is essentially useless: it relies on additional packages to describe the structure of documents, to bind keys, to maintain a history of cancellations. Although using this system is really a little difficult, it has found application in programs where you need to configure things that are not supported in classic editors.
The ProseMirror extension mechanism is relatively straightforward. When creating an editor, a single array of connected objects is specified in the client code. Each of these plug-in objects can affect various aspects of the editor and do things such as adding bits of status data or handling interface events.
All these aspects were designed to work with an ordered array of configuration values, using one of the strategies described above. For example, when you need to specify a lot of dictionaries with values, the priority of following extension instances for key binding depends on the order in which you specify these instances. The first extension for key binding, knowing what to do with this keystroke, gets it processed.
Usually, such a mechanism turns out to be quite powerful, and they can turn it to their advantage. But sooner or later, the extension system reaches such complexity that it becomes inconvenient to use it.
- If the plugin has many effects, then you can only hope that either they all need the same priority relative to other plugins, or you have to break them into smaller plugins in order to properly arrange their order.
- In general, organizing plugins becomes very sensitive, because the end user is not always clear which plugins can interfere with which other plugins if they get a higher priority. Errors made in such cases usually occur only at runtime, when a specific opportunity is used, so they are easy to overlook.
- If plugins are built on the basis of other plugins, then this fact needs to be documented and it is hoped that users will not forget to include the appropriate dependencies (at that ordering step, when necessary).
CodeMirror version 6 is a rewritten version of the code editor of the same name. In this project, I try to develop a modular approach. To do this, I need a more expressive extension system. Let's discuss some of the challenges that we had to deal with when designing such a system.
Ordering
It’s easy to design a system that gives you complete control over the ordering of extensions. However, it is much more difficult to design a system with which it will be pleasant to work with and which, at the same time, will allow you to combine the code of various extensions without numerous interventions from the “now watch your hands” category.
When it comes to ordering, it happens, I really want to resort to work with priority values. As an example, the CSS
z-index
property indicates the number of the position occupied by this element in the depth of the stack.
As you can see on the example of ridiculously large
z-index
values that are sometimes found in style sheets, this way of indicating priority is problematic. The module itself does not know what priority values other modules have. Options are just dots in the middle of an undifferentiated numerical range. You can set a huge (or deeply negative) value to try to reach the far edges of this spectrum, but the rest of the work comes down to fortune telling.
The situation can be improved a little if you define a limited set of clearly defined priority categories, so that extensions can characterize the general “level” of their priority. In addition, you will need a certain way to break the links within the categories.
Grouping and Deduplication
As I mentioned above, as soon as you start to rely seriously on extensions, a situation may arise when some extensions will use others when working. If you manage dependencies manually, then this approach does not scale well; therefore, it would be nice if you could pull up a group of extensions at the same time.
However, this approach not only further aggravates the priority problem, but also introduces another problem: many other extensions may depend on a particular extension, and if the extensions are presented as values, it may well be that the same extension will be loaded multiple times . For some types of extensions, such as event handlers, this is normal. In other cases, such as with a cancellation history and a tooltip library, this approach will be wasteful and may even break everything.
So, if we allow the layout of extensions, this introduces into our system a certain additional complexity associated with dependency management. You must be able to recognize such extensions that should not be duplicated, and download them exactly one at a time.
However, since in most cases extensions can be configured, and therefore not all instances of the same extension will be exactly the same, we cannot just take one instance and work with it. We will have to consider some meaningful merger of such instances (or report an error if the merge of interest to us is impossible).
Project
Here I will describe in general terms what we are doing in CodeMirror 6. This is just a sketch, not a Failed Solution. It is possible that this system will develop further when the library stabilizes.
The main primitive used in this approach is called behavior. Behaviors are just those features that you can build on with extensions, specifying values for them. An example is the behavior of a state field where, with the help of extensions, you can add new fields, providing a description of the field. Another example is the behavior of event handlers in a browser; in this case, with the help of extensions, we can add our own handlers.
From the point of view of the consumer of behaviors, the behaviors themselves, configured in a certain way in a particular instance of the editor, give an ordered sequence of values, and those values that come before have a higher priority. Each behavior has a type, and the values provided for it must match that type.
The behavior is represented as a value used both to declare an instance of the behavior and to access the values that the behavior has. There are a number of built-in behaviors in the library, but external code can define its own behaviors. For example, in the extension defining the interval between line numbers, behavior can be defined that allows another code to add additional markers in this interval.
An extension is a value that can be used when configuring the editor. An array of such values is passed during initialization. Each extension is allowed in zero or more behaviors.
Such a simple extension can be considered an instance of behavior. If we specify a value for the behavior, then the code returns us the value of the extension that generates this behavior.
A sequence of extensions can also be grouped into a single extension. For example, in the editor’s configuration for working with a specific programming language, you can pull up several other extensions - for example, grammar for parsing and highlighting text, information about the necessary indentation, an autocompletion source that will correctly display prompts for completing lines in this language. Thus, you can make a single language extension in which we simply collect all these corresponding extensions and group them together, resulting in a single value.
When creating a simple version of such a system, we could stop at this by simply aligning all the nested extensions in one array of behavior extensions. Then they could be grouped by type of behavior and then build ordered sequences of values of behavior.
However, it remains to deal with deduplication and provide better control over ordering.
The values of extensions related to the third type, unique extensions, just help to achieve deduplication. Extensions that should not be instantiated twice in the same editor are of this kind. To determine such an extension, you need to specify the spec-type , that is, the type of configuration value expected by the extension constructor, and also specify an instantiation function that takes an array of such specified values and returns the extension.
Unique extensions complicate the process of resolving a set of extensions into a set of behaviors. If there are unique extensions in the aligned set of extensions, then the resolving mechanism should select the type of unique extension, collect all its instances and call the corresponding instantiation function along with the specifications, and then replace them all with the result (in a single copy).
(There’s another catch: they must resolve in the correct order. If you first allow the unique extension X, but then get another X as a result of the resolution, then this will be wrong, since all instances of X must be put together. Since the instantiation function expansion is clean, the system copes with this situation by trial and error, restarting the process and recording information about what it was possible to study, being in this situation.)
Finally, you need to resolve the issue with the rules of the sequence. The basic approach remains the same: maintain the order in which extensions were proposed. Compound extensions align in the same order at the point where they occur. The result of resolving a unique extension is inserted when it is first turned on.
However, extensions can relate some of their sub extensions to categories that have a different priority. The system provides for four such categories: fallback (takes effect after other things happen), default (default), extend (higher priority than the bulk) and override (probably should go first). In practice, sorting is done first by category, and then by starting position.
So, a key binding extension, which has a low priority, and an event handler with a normal priority, it’s fashionable to get a compound extension based on the result of a key binding extension (not requiring knowing what behavior it consists of) with a fallback priority level and from an instance with the behavior of the event handler.
This approach, which allows you to combine extensions without thinking about what they do “inside,” seems to be a great achievement. The extensions we modeled earlier in this article include two parsing systems that exhibit the same behavior at the syntax level, syntax highlighting service, smart indentation service, cancellation history, line-spacing service, auto-closing brackets, key binding and multiple selection - all works good.
There are several new concepts that the user must learn in order to use this system. In addition, working with such a system is indeed a bit more complicated than with traditional imperative systems adopted in the JavaScript community (we call a method to add / remove an effect). However, if the extensions are properly arranged, then the benefits of this outweigh the associated costs.
All Articles