Using JavaScript modules in production: current state of affairs. Part 1
Two years ago, I wrote about a technique that is now commonly called the module / nomodule pattern. Its application allows you to write JavaScript code using the capabilities of ES2015 +, and then use bundlers and transpilers to create two versions of the code base. One of them contains modern syntax (it is loaded using a structure like
, and the second is ES5 syntax (it is loaded using
). The module / nomodule pattern allows sending to browsers that support modules, much less code than browsers that do not support this feature, now this pattern is supported by most web frameworks and command line tools.
Previously, even considering the ability to send modern JavaScript code to production, and even though most browsers supported modules, I recommended collecting code in bundles.
Why? Mostly because I had the feeling that loading modules into the browser was slow. Even though recent protocols, such as HTTP / 2, theoretically supported the efficient loading of multiple files, all performance studies at the time concluded that using bundlers is still more efficient than using modules.
But it must be admitted that those studies were incomplete. Test cases using the modules that were studied in them consisted of non-optimized and non-minimized source code files that were deployed in production. There were no comparisons of the optimized bundle with modules with the optimized classic script.
However, to be honest, then there was no optimal way to deploy the modules. But now, thanks to some modern improvements in bundler technologies, it is possible to deploy production code in the form of ES2015 modules using both static and dynamic import commands, and at the same time receive higher performance than can be achieved using the available options, in which modules are not used.
It should be noted that on the site on which the original material is published, the first part of the translation of which we publish today, the modules have been used in production for several months.
Many people with whom I have talked completely reject modules, not even considering them as one of the options for large-scale production applications. Many of them cite the very study that I have already mentioned. Namely, that part of it, which states that the modules should not be used in production, unless we are talking about “small web applications that include less than 100 modules that differ in a relatively“ small ”dependency tree (that is - one whose depth does not exceed 5 levels). ”
If you have ever looked into the
directory of any of your projects, then you probably know that even a small application can easily have more than 100 dependency modules. I want to offer you a look at how many modules are available in some of the most popular npm packages.
This is where the main misconception regarding modules is rooted. Programmers believe that when it comes to using modules in production, they have only two options. The first is to deploy all the source code in its existing form (including the
directory). The second is not to use modules at all.
However, if you look closely at the recommendations from the study cited above, you will find that there is nothing to say that loading modules is slower than loading regular scripts. It does not say that modules should not be used at all. It just talks about the fact that if someone deploys hundreds of uninfected module files in production, Chrome won’t be able to load them as fast as a single minified bundle. As a result, the study advises continuing to use bundlers, compilers, and minifiers.
But you know what? The fact is that you can use all this and use modules in production.
In fact, modules are a format that we should strive to convert to, because browsers already know how to load modules (and browsers that cannot do this can load a fallback using the nomodule mechanism). If you look at the code that the most popular bundlers generate, you will find many template fragments whose purpose is only to dynamically load other code and manage dependencies. But all this will not be necessary if we just use the modules and expressions
and
.
Fortunately, at least one of the popular modern bundlers ( Rollup ) supports modules in the form of output data . This means that you can process the code with a bundler and deploy modules in the production (without using template fragments to load the code). And, since Rollup has an excellent implementation of the tree-shaking algorithm (the best I've seen in bundlers), building programs in the form of modules using Rollup allows you to get code that is smaller than the size of a similar code obtained when applying other mechanisms available today.
It should be noted that they plan to add support for modules in the next version of Parcel. Webpack does not yet support modules as an output format, but there you have it - discussions that focus on this issue.
Another misconception regarding modules is that some people believe that modules can only be used if 100% of the project dependencies use modules. Unfortunately (I believe it is very unfortunate), most npm packages are still being prepared for publication using the CommonJS format (some modules, even those written using ES2015 features, are translated into CommonJS format before being published to npm)!
Here, again, I want to note that Rollup has a plug-in ( rollup-plugin-commonjs ) that takes input source code written using CommonJS and converts it to ES2015 code. Definitely, it would be better if the dependency format used from the very beginning uses the ES2015 module format. But if some dependencies are not, this does not prevent you from deploying projects using modules in production.
In the following parts of this article, I’m going to show you how I collect projects in bundles that use modules (including the use of dynamic imports and code separation), I’m going to talk about why such solutions are usually more productive than classical scripts, and show how they work with browsers that do not support modules.
Building code for production is always an attempt to balance the pros and cons of various solutions. On the one hand, the developer wants his code to load and execute as quickly as possible. On the other hand, he does not want to download code that will not be used by users of the project.
In addition, developers need confidence that their code is best suited for caching. The big problem of code bundling is that any change to the code, even one changed line, leads to invalidation of the cache of the entire bundle. If you deploy an application consisting of thousands of small modules (presented exactly in the form in which they are present in the source code), then you can safely make minor changes to the code and at the same time know that most of the application code will be cached . But, as I already said, such an approach to development can probably mean that loading the code on the first visit to the resource may take longer than when using more traditional approaches.
As a result, we face a difficult task, which is to find the right approach to breaking the bundles into parts. We need to strike the right balance between the speed of loading materials and their long-term caching.
Most bundlers, by default, use code splitting techniques based on dynamic import commands. But I would say that dividing code only with a focus on dynamic import does not allow breaking it into sufficiently small fragments. This is especially true for sites with many returning users (that is, in situations where caching is important).
I believe that code should be broken into as small fragments as possible. It is worth reducing the size of fragments until their number grows so much that it will affect the download speed of the project. And although I definitely recommend that everyone do their own analysis of the situation, if you believe the approximate calculations made in the study I mentioned, when loading less than 100 modules, there is no noticeable slowdown in loading. A separate study on HTTP / 2 performance did not reveal a noticeable project slowdown when downloading less than 50 files. There, however, they tested only options in which the number of files was 1, 6, 50, and 1000. As a result, probably 100 files are a value that you can easily navigate to without fear of losing in download speed.
So, what is the best way to aggressively, but not too aggressively split the code into parts? In addition to splitting code based on dynamic import commands, I would advise you to take a closer look at splitting code into npm packages. With this approach, what is imported into the project from the
folder falls into a separate fragment of the finished code based on the package name.
I said above that some of the modern capabilities of bundlers make it possible to organize a high-performance scheme for deploying projects based on modules. What I was talking about is represented by two new Rollup features. The first is automatic code separation through dynamic
commands (added in v1.0.0 ). The second option is manual code separation performed by the program based on the
option (added in v1.11.0 ).
Thanks to these two features, it is now very easy to configure the build process, in which code is split at the package level.
Here is an example configuration that uses the
option, thanks to which each module imported from
falls into a separate piece of code whose name corresponds to the name of the package (technically, the name of the package directory in the
folder):
The
option accepts a function that accepts, as a single argument, the path to the module file. This function can return a string name. What it returns will point to a fragment of the assembly to which the current module should be added. If the function does not return anything, then the module will be added to the fragment used by default.
Consider an application that imports the
,
and
from the
package. If you apply the above configuration when building this application, then each of these modules (as well as each
module imported by these modules) will be placed in a single output file with a name like
(here
is unique module file hash in the
fragment).
At the end of the file, you will see an export expression like the following. Please note that this expression contains only export commands for modules added to the fragment, and not all
modules.
Then, if the code in any of the other fragments uses these
modules (perhaps only the
method), in these fragments, at the top of them, there will be an import expression that looks like this:
Hopefully this example clarified the question of how manual code separation works in Rollup. In addition, I think that the results of code separation using the
and
expressions are much easier to read and understand than the code of fragments, the formation of which used non-standard mechanisms that are used only in some bundler.
For example, it’s very difficult to figure out what is going on in the next file. This is the output of one of my old projects that used webpack to split code. Almost everything in this code is not needed in browsers that support modules.
As I already said, I believe that code-level separation at the package level usually allows the developer to get into a favorable position when code separation is aggressive, but not too aggressive.
Of course, if your application imports modules from hundreds of different npm packages, you can still be in a situation where the browser cannot load them all effectively.
However, if you really have many npm dependencies, you should not completely abandon this strategy for now. Remember that you probably won’t download all npm dependencies on every page. Therefore, it is important to find out how many dependencies actually load.
Nevertheless, I am sure that there are some real applications that have so many npm dependencies that these dependencies simply cannot be represented as separate fragments. If your project is just that - I would recommend that you look for a way to group packages where the code with high probability can change at the same time (like
and
) since fragment cache
with these packages will also be executed at the same time. Later I will show an example in which all React dependencies are grouped in the same fragment .
To be continued…
Dear readers! How do you approach the problem of code separation in your projects?
<script type="module">
, and the second is ES5 syntax (it is loaded using
<script nomodule>
). The module / nomodule pattern allows sending to browsers that support modules, much less code than browsers that do not support this feature, now this pattern is supported by most web frameworks and command line tools.
Previously, even considering the ability to send modern JavaScript code to production, and even though most browsers supported modules, I recommended collecting code in bundles.
Why? Mostly because I had the feeling that loading modules into the browser was slow. Even though recent protocols, such as HTTP / 2, theoretically supported the efficient loading of multiple files, all performance studies at the time concluded that using bundlers is still more efficient than using modules.
But it must be admitted that those studies were incomplete. Test cases using the modules that were studied in them consisted of non-optimized and non-minimized source code files that were deployed in production. There were no comparisons of the optimized bundle with modules with the optimized classic script.
However, to be honest, then there was no optimal way to deploy the modules. But now, thanks to some modern improvements in bundler technologies, it is possible to deploy production code in the form of ES2015 modules using both static and dynamic import commands, and at the same time receive higher performance than can be achieved using the available options, in which modules are not used.
It should be noted that on the site on which the original material is published, the first part of the translation of which we publish today, the modules have been used in production for several months.
Misconceptions about modules
Many people with whom I have talked completely reject modules, not even considering them as one of the options for large-scale production applications. Many of them cite the very study that I have already mentioned. Namely, that part of it, which states that the modules should not be used in production, unless we are talking about “small web applications that include less than 100 modules that differ in a relatively“ small ”dependency tree (that is - one whose depth does not exceed 5 levels). ”
If you have ever looked into the
node_modules
directory of any of your projects, then you probably know that even a small application can easily have more than 100 dependency modules. I want to offer you a look at how many modules are available in some of the most popular npm packages.
This is where the main misconception regarding modules is rooted. Programmers believe that when it comes to using modules in production, they have only two options. The first is to deploy all the source code in its existing form (including the
node_modules
directory). The second is not to use modules at all.
However, if you look closely at the recommendations from the study cited above, you will find that there is nothing to say that loading modules is slower than loading regular scripts. It does not say that modules should not be used at all. It just talks about the fact that if someone deploys hundreds of uninfected module files in production, Chrome won’t be able to load them as fast as a single minified bundle. As a result, the study advises continuing to use bundlers, compilers, and minifiers.
But you know what? The fact is that you can use all this and use modules in production.
In fact, modules are a format that we should strive to convert to, because browsers already know how to load modules (and browsers that cannot do this can load a fallback using the nomodule mechanism). If you look at the code that the most popular bundlers generate, you will find many template fragments whose purpose is only to dynamically load other code and manage dependencies. But all this will not be necessary if we just use the modules and expressions
import
and
export
.
Fortunately, at least one of the popular modern bundlers ( Rollup ) supports modules in the form of output data . This means that you can process the code with a bundler and deploy modules in the production (without using template fragments to load the code). And, since Rollup has an excellent implementation of the tree-shaking algorithm (the best I've seen in bundlers), building programs in the form of modules using Rollup allows you to get code that is smaller than the size of a similar code obtained when applying other mechanisms available today.
It should be noted that they plan to add support for modules in the next version of Parcel. Webpack does not yet support modules as an output format, but there you have it - discussions that focus on this issue.
Another misconception regarding modules is that some people believe that modules can only be used if 100% of the project dependencies use modules. Unfortunately (I believe it is very unfortunate), most npm packages are still being prepared for publication using the CommonJS format (some modules, even those written using ES2015 features, are translated into CommonJS format before being published to npm)!
Here, again, I want to note that Rollup has a plug-in ( rollup-plugin-commonjs ) that takes input source code written using CommonJS and converts it to ES2015 code. Definitely, it would be better if the dependency format used from the very beginning uses the ES2015 module format. But if some dependencies are not, this does not prevent you from deploying projects using modules in production.
In the following parts of this article, I’m going to show you how I collect projects in bundles that use modules (including the use of dynamic imports and code separation), I’m going to talk about why such solutions are usually more productive than classical scripts, and show how they work with browsers that do not support modules.
Optimal Code Build Strategy
Building code for production is always an attempt to balance the pros and cons of various solutions. On the one hand, the developer wants his code to load and execute as quickly as possible. On the other hand, he does not want to download code that will not be used by users of the project.
In addition, developers need confidence that their code is best suited for caching. The big problem of code bundling is that any change to the code, even one changed line, leads to invalidation of the cache of the entire bundle. If you deploy an application consisting of thousands of small modules (presented exactly in the form in which they are present in the source code), then you can safely make minor changes to the code and at the same time know that most of the application code will be cached . But, as I already said, such an approach to development can probably mean that loading the code on the first visit to the resource may take longer than when using more traditional approaches.
As a result, we face a difficult task, which is to find the right approach to breaking the bundles into parts. We need to strike the right balance between the speed of loading materials and their long-term caching.
Most bundlers, by default, use code splitting techniques based on dynamic import commands. But I would say that dividing code only with a focus on dynamic import does not allow breaking it into sufficiently small fragments. This is especially true for sites with many returning users (that is, in situations where caching is important).
I believe that code should be broken into as small fragments as possible. It is worth reducing the size of fragments until their number grows so much that it will affect the download speed of the project. And although I definitely recommend that everyone do their own analysis of the situation, if you believe the approximate calculations made in the study I mentioned, when loading less than 100 modules, there is no noticeable slowdown in loading. A separate study on HTTP / 2 performance did not reveal a noticeable project slowdown when downloading less than 50 files. There, however, they tested only options in which the number of files was 1, 6, 50, and 1000. As a result, probably 100 files are a value that you can easily navigate to without fear of losing in download speed.
So, what is the best way to aggressively, but not too aggressively split the code into parts? In addition to splitting code based on dynamic import commands, I would advise you to take a closer look at splitting code into npm packages. With this approach, what is imported into the project from the
node_modules
folder falls into a separate fragment of the finished code based on the package name.
Package Separation
I said above that some of the modern capabilities of bundlers make it possible to organize a high-performance scheme for deploying projects based on modules. What I was talking about is represented by two new Rollup features. The first is automatic code separation through dynamic
import()
commands (added in v1.0.0 ). The second option is manual code separation performed by the program based on the
manualChunks
option (added in v1.11.0 ).
Thanks to these two features, it is now very easy to configure the build process, in which code is split at the package level.
Here is an example configuration that uses the
manualChunks
option, thanks to which each module imported from
node_modules
falls into a separate piece of code whose name corresponds to the name of the package (technically, the name of the package directory in the
node_modules
folder):
export default { input: { main: 'src/main.mjs', }, output: { dir: 'build', format: 'esm', entryFileNames: '[name].[hash].mjs', }, manualChunks(id) { if (id.includes('node_modules')) { // , `node_modules`. // - , . const dirs = id.split(path.sep); return dirs[dirs.lastIndexOf('node_modules') + 1]; } }, }
The
manualChunk
option accepts a function that accepts, as a single argument, the path to the module file. This function can return a string name. What it returns will point to a fragment of the assembly to which the current module should be added. If the function does not return anything, then the module will be added to the fragment used by default.
Consider an application that imports the
cloneDeep()
,
debounce()
and
find()
lodash-es
from the
lodash-es
package. If you apply the above configuration when building this application, then each of these modules (as well as each
lodash
module imported by these modules) will be placed in a single output file with a name like
npm.lodash-es.XXXX.mjs
(here
XXXX
is unique module file hash in the
lodash-es
fragment).
At the end of the file, you will see an export expression like the following. Please note that this expression contains only export commands for modules added to the fragment, and not all
lodash
modules.
export {cloneDeep, debounce, find};
Then, if the code in any of the other fragments uses these
lodash
modules (perhaps only the
debounce()
method), in these fragments, at the top of them, there will be an import expression that looks like this:
import {debounce} from './npm.lodash.XXXX.mjs';
Hopefully this example clarified the question of how manual code separation works in Rollup. In addition, I think that the results of code separation using the
import
and
export
expressions are much easier to read and understand than the code of fragments, the formation of which used non-standard mechanisms that are used only in some bundler.
For example, it’s very difficult to figure out what is going on in the next file. This is the output of one of my old projects that used webpack to split code. Almost everything in this code is not needed in browsers that support modules.
(window["webpackJsonp"] = window["webpackJsonp"] || []).push([["import1"],{ /***/ "tLzr": /*!*********************************!*\ !*** ./app/scripts/import-1.js ***! \*********************************/ /*! exports provided: import1 */ /***/ (function(module, __webpack_exports__, __webpack_require__) { "use strict"; __webpack_require__.r(__webpack_exports__); /* harmony export (binding) */ __webpack_require__.d(__webpack_exports__, "import1", function() { return import1; }); /* harmony import */ var _dep_1__WEBPACK_IMPORTED_MODULE_0__ = __webpack_require__(/*! ./dep-1 */ "6xPP"); const import1 = "imported: " + _dep_1__WEBPACK_IMPORTED_MODULE_0__["dep1"]; /***/ }) }]);
What if there are hundreds of npm dependencies?
As I already said, I believe that code-level separation at the package level usually allows the developer to get into a favorable position when code separation is aggressive, but not too aggressive.
Of course, if your application imports modules from hundreds of different npm packages, you can still be in a situation where the browser cannot load them all effectively.
However, if you really have many npm dependencies, you should not completely abandon this strategy for now. Remember that you probably won’t download all npm dependencies on every page. Therefore, it is important to find out how many dependencies actually load.
Nevertheless, I am sure that there are some real applications that have so many npm dependencies that these dependencies simply cannot be represented as separate fragments. If your project is just that - I would recommend that you look for a way to group packages where the code with high probability can change at the same time (like
react
and
react-dom
) since fragment cache
react-dom
with these packages will also be executed at the same time. Later I will show an example in which all React dependencies are grouped in the same fragment .
To be continued…
Dear readers! How do you approach the problem of code separation in your projects?
All Articles