How ES Module Shims became a Production Import Maps Polyfill

Refer

Source: How ES Module Shims became a Production Import Maps Polyfill - 4 April 2022

Author: Guy Bedford

Translator: SenaoXi

Copyright Statement

Translation and Republication Notice:

This translation is provided for educational and informational purposes only. All intellectual property rights, including copyright, remain with the original author and/or publisher. This translation maintains the integrity of the original content while making it accessible to chinese readers.

Modifications Disclosure:

• This is a complete and faithful translation of the original content with no substantive modifications.
• This translation includes minor adaptations to improve clarity for chinese readers while preserving all essential information and viewpoints.
• Sections marked with [†] contain supplementary explanations added by the translator to provide cultural or technical context.

Rights Reservation:

If you are the copyright holder and believe this translation exceeds fair use guidelines, please contact us at email. We are committed to respecting intellectual property rights and will promptly address any legitimate concerns.

ES Module Shims was initially created as a polyfill to enable the use of new native module features (such as import maps) before browsers officially supported them, suitable for rapid development or simple production workflows.

Over time, ES Module Shims has evolved into a highly optimized, production-ready pluggable polyfill for import maps.

With import maps now having 70% browser support among users, and Firefox just announcing support for import maps, it's a good time to share the story of ES Module Shims and its implementation details.

This article first introduces the background of the project architecture, then describes how the polyfill mode came to be, provides comprehensive performance benchmarks, and finally looks at the future direction of the project.

Thanks to Basecamp for seeing the potential in this project and sponsoring new performance research for ES Module Shims, and to Rich Harris for the initial inspiration for this project.

Loader Architecture

The core of ES Module Shims is a module loader. While script execution can be easily achieved with a simple single-parameter eval() function, setting up a module loader is much more complex. This is because modular execution requires providing module source code, supporting the discovery and resolution of dependencies from the source code, then asynchronously providing the source code of these dependencies, and repeating the above parsing process.

Unfortunately, there is no modular execution API on the Web. When ES Module Shims was created in 2018, all major browsers supported <script type="module">, but support for other new features like import() or import.meta.url was inconsistent.

To implement features like import maps, custom module dependency resolution is needed, which requires some kind of loader hook to achieve this.

The initial problem faced by ES Module Shims was: How to build an arbitrary module loader with custom dependency and source text resolution on top of <script type="module">?

This is not obvious to implement, but can be achieved through two technical approaches:

1. A small & fast JS module lexer.
2. Source customization via Blob URLs.

1. A small & fast JS module lexer

Given any source text string source, the first thing to determine is the location of the import statement.

For example, for this source code:

const source = `import { dep } from 'dep';
console.log(dep, mapped);`;

We need to know that there is an import 'dep' at position [21, 28] in the source string.

If we can determine the location of the import, we can parse these imports and then rewrite the source string using the same offset by string manipulation:

const transformed =
  source.slice(0, 21) + '/dep-resolved.js' + source.slice(28);

The parsed result:

import { dep } from '/dep-resolved.js';
console.log(dep, mapped);

Then the question is, how do we write such a lexer to get the import statement location information?

How to write an imports lexer?

If js's specification had required imports to only appear before other types of js statements, then it would be easy. But the problem is that the following code is also valid js syntax:

const someText = `import 'not an import'`;

// import 'also not an import';

'asdf'.match(/import 'you guessed it'/);

import 'oh dear';

The above examples might seem like silly edge cases, but when providing a complete solution, these edge cases must be considered.

While various regular expressions related to import can be used to get the import statement location information, this is not a reliable foundation for this project. Because the problem with using regular expressions is that they cannot consistently handle language syntax edge cases —— there will always be some modules that will fail to parse due to edge cases, breaking the entire reliable implementation of module parsing process.

Another option is to use a parser, but even the smallest parser at the time was over 100KB, and this would bring significant performance overhead for this use case.

Rich-Harris-Inspired Magic

When Rich Harris released Shimport (I imagine he created this project in a weekend or two, and then started building Svelte the next weekend), Shimport showed dynamic dependency analysis in the browser using a small and efficient javascript lexer, suitable for rapid development or simple production workflows, which I had never thought possible.

Shimport used a custom lexer to dynamically rewrite ES modules in the browser, to support browsers without ES modules support.

When Shimport re-parsed all imports and exports, compared to just parsing imports and rewriting their internal string, it seemed simpler, and dynamic module binding would naturally be supported by using native modules.

When discussing how to handle the complexity of js language, without deeper parser knowledge, he shared a key technical point of handling one of the main issues with js lexer —— the division regular expression ambiguity.

Division Regular Expression Ambiguity

For most js lexers, the rules are quite simple —— for strings, read from the first " to the last, handle escapes. Comments (/* and //) also follow simple rules. Template expressions (`..${`..`}..`) have some nested needs through nested parser functions, and regular expressions also have some minor cases to check. Apart from that, basically just match open brackets [, (, and { with their corresponding close brackets, and no further parser context information is needed to complete parsing.

Then, when all brackets are closed, we know we are at the top level, and we can start parsing imports using their syntax parser rules to get a small module lexer.

However, once / appears, there is a lexer ambiguity problem. For example:

while (items.length) /regexp/.test(items.pop()) || error();

In the above, the / is distinguished as regular expression rather than division because the brackets are the end of the while statement, but knowing this requires full understanding of the while statement parser state.

So the trick is to create a minimal lexer context stack, matching open brackets(() and close brackets()), and including enough state information to handle main ambiguity cases like above.

Determining whether the parser is in an import() expression position (otherwise it might conflict with class method definition) can be handled using similar methods, by brackets and brackets stack associated minimal lexer context to judge.

Initially implemented in ES Module Shims as a native js embedded lexer, later this implementation was converted to C language implementation, and obtained more performance advantages (actually mainly cold start advantages, V8 is not free!) through WebAssembly compilation (C compiled to the smallest Wasm output one). It became the es-module-lexer project.

To illustrate the JS lexer performance, the entire project is a 4.5kb js file, which can analyze 1MB js code in less than 10ms on most desktop devices, with performance basically linear growth. es-module-lexer has become a popular npm package, with weekly downloads exceeding 750 million times. Sharing a good solution for a common problem is valuable.

2. Source customization via Blob URLs

Blob URL can be used with import() to execute arbitrary module source code:

const blob = new Blob(['export const itsAModule = true'], {
  type: 'text/javascript'
});

const blobUrl = URL.createObjectURL(blob);

const { itsAModule } = await import(blobUrl);
console.log(itsAModule); // true!

In browsers without dynamic import support, the same effect can be achieved by dynamically injecting <script type="module">import 'blob:...'</script> tags into the DOM.

Once everything is ready, we need to consider processing the entire module dependency graph, assuming we have an application like this:

<script type="importmap">
  {
    "imports": {
      "dep": "/dep.js"
    }
  }
</script>
<script type="module" src="./app.js"></script>

Where app.js contains:

import depA from 'dep';

export default {};

The import dep dependency will be resolved through import map to /dep.js.

If we rewrite or customize the /dep.js source code, we will get a dep blob URL, then we must write it into the corresponding import position in the above /app.js source code. Then we create a blob URL from the transformed app.js source code, and finally dynamically import that blob URL.

The final import('blob:site.com/72ac72b2-8106') completely contains the entire dependency graph execution of app.js, making ES Module Shims a fully customizable loader on top of native modules, fully supporting live bindings and cycle dependency processing (except for some minor modifications to live bindings in cycles).

Shim Mode

The initial implementation was to use custom module-shim and importmap-shim script types:

<script type="importmap-shim">
  {
    "imports": {
      "dep": "/packages/dep/main.js"
    }
  }
</script>
<script type="module-shim">
  import dep from 'dep';
  console.log(dep);
</script>

This ensures that it does not conflict with native loaders, and ES Module Shims uses fetch() to handle dependencies, then lazily rewrites source code through es-module-lexer and inlines them as Blob URLs.

Polyfill Mode

2021.03, Chrome released flagless support for import maps. Soon after, using the complete native import maps workflow became attractive, and ES Module Shims was applied only when needed in browsers without import maps.

When using import maps in browsers without support, a static error is thrown:

Because this is a static error (occurring at link stage, not dynamic error at execution time), no module is executed when this error occurs. Therefore, ES Module Shims will only execute module dependency graph through its loader when native execution fails, and because this is a link stage decision, there will be no repeated execution module risk. Depending on the browser, fetch cache can also be shared.

The polyfill mode of ES Module Shims then became a way to re-execute static failure. When you statically use the feature you want to fill, once the browser does not support import maps, the browser will always throw a static error, and ES Module Shims will detect this and execute module through its loader to implement polyfill.

In addition, in browsers with full import maps support (or current new popular baseline module features), even without ES Module Shims analyzing source code, baseline pass-through can be achieved.

The polyfill mode processing includes the following steps:

1. Run feature detection for import maps and related module features.
2. If the browser supports all modern features, no further action is needed.
3. Track analysis module source code through loader fetch and lexer(es-module-lexer) to determine if there is a static throw scenario due to native new features.
4. If the source code analysis determines that the module graph is statically thrown, load module dependency graph in ES Module Shims loader.

69% user browsers already support import maps, ES Module Shims only do feature detection, basically zero work, so the polyfill mode eventually became a high-performance method to fill module features for old browsers.

The simplest workflow is to use import map from the beginning:

<script
  async
  src="https://ga.jspm.io/npm:es-module-shims@1.5.4/dist/es-module-shims.js"
></script>
<script type="importmap">
  {
    "imports": {
      "app": "/app/main.js"
    }
  }
</script>
<script type="module">
  import 'app';
</script>

By directly using import map, an error will occur immediately in browsers without native support, avoiding unnecessary browser processing, and ensuring clean polyfill switch.

Performance

To verify performance, Basecamp sponsored some benchmark and optimization work for ES Module Shims.

The performance goal of ES Module Shims is to verify:

1. Baseline Passthrough: Verify that in browsers with import maps support, load ES Module Shims polyfill does not cause any unnecessary slowdown and matches native performance.
2. Polyfill Performance: Quantify polyfill load cost in browsers without import maps.
3. Import maps Performance: Investigate performance of large import maps with many modules.

Benchmark Setup

All performance tests use Preact for simple component rendering, and include page full load time.

Each sample n contains component and Preact load and execution, about 10KB js code load and interaction with DOM.

When n = 100, it means load and execute 1MB code (about 10KB each sample). All scenarios below are based on uncached performance detection, and use different network environments according to each case.

Benchmark results and source code can be found on ES Module Shims repo. Use Tachometer to execute multiple runs. Tests executed on standard desktop.

Baseline Passthrough Performance

In this case, the goal is verify that for about 70% support import maps users, load ES Module Shims polyfill does not cause any unnecessary slowdown and matches native performance.

To verify this, benchmark compares using import maps load n samples of Preact + component rendering in Chrome, with and without ES Module Shims script tags.

Benchmark includes full time to load ES Module Shims into browser and run feature detection.

Chrome native import maps load time, page with (orange) and without (blue) ES Module Shims, for load n samples change.

We can see that page with ES Module Shims causes slight additional load time, average about 6.5ms, this is the time for ES Module Shims initialization and running feature detection.

In most cases, performance is the same, corresponding to application native pass-through, polyfill does not participate at all.

Baseline Passthrough with Throttling

As we start limiting network, ES Module Shims bandwidth cost should start showing, because it has 12KB network download size.

Throttling set to 750KB/s and 25ms RTT.

Chrome native import maps throttled network load time, page with (orange) and without (blue) ES Module Shims, for load n samples change.

Throttling introduces some noise, but on average, additional load time is about 10ms, expected 12KB js load time on 750KB/s network about 15ms (remember everything is parallel, so execution time fills network gap).

Therefore, we can conclude that for about 70% support import maps users, polyfill impact on performance is mostly negligible, corresponding to 12KB download and some initialization time, usually not more than about 15ms.

Polyfill Performance

To test polyfill overhead when ES Module Shims fully participates, we cannot turn off native module support in Chrome, but we can use the following assumption:

Using and not using small import map load module dependency graph cost should be roughly similar.

Based on this assumption, to compare native load performance and polyfill participation load performance, we can use Firefox browser to ensure polyfill participation scenario, compare using import maps and naked specifier (currently not supported import maps in Firefox, so polyfill will be enabled) situation with directly importing native supported URL (page completely without ES Module Shims).

Firefox native module load time, without using import maps and ES Module Shims (blue), using import maps and ES Module Shims and enable polyfill (orange) compared for load n samples change.

Compared to native, ES Module Shims polyfill layer cost shows obvious linear slowdown, because all code is executed through ES Module Shims loader in import maps case.

Linear correlation shows polyfill cost average 1.59 times slower than native load. 100 modules 1MB code native load and execution need 220ms, while using ES Module Shims polyfill need 320ms (including all additional polyfill load and initialization time).

Polyfill with Throttling

Similarly, check throttling result for 750KB/s and 5ms RTT:

Firefox native module throttled network load time, without using import maps and ES Module Shims (blue), using import maps and ES Module Shims and enable polyfill (orange) compared for load n samples change.

From data, on throttled connection average slowdown is about 1.14 times. 100 modules 1MB code native load need 692ms, while using throttled polyfill need 744ms. With throttling, polyfill overhead reduces because network becomes bottleneck rather than polyfill.

Import Maps Performance

The last question is, whether large import maps (e.g., hundreds of lines) will slow down page load speed because this scenario is located on the critical load path.

To study this, we replaced the previous simple two-line import map with a new import map, which generates a line of import map for each sample case. Therefore, n = 10 corresponds to must use 20 different import map lines, n = 100 when increased to 200 lines.

We use throttled connection of 750KB/s and 25ms RTT, as this is primarily a network issue rather than a CPU issue.

Chrome native module throttled network load time, comparing import map with only two entries (blue) versus import map with separate entry pairs for each n sample (orange), for different n samples change.

The slowdown is very slight, less than 10ms for n = 100, corresponding to larger import map only adding extra download time, proving that the performance cost of larger import maps is negligible.

Polyfill Import Maps Performance

Finally, as before, we compare but for Firefox and ES Module Shims, we expect roughly the same dynamics, with no surprises. Comparing import map with two entries versus import map with two import map entries for each sample n, both loaded using ES Module Shims polyfill:

Firefox and ES Module Shims import map throttled network load time, comparing import map with only two entries (blue) versus import map with separate entry pairs for each n sample (orange), for different n samples change.

The fluctuation at n = 30 may be a product of the throttling process itself.

As expected, on throttled connection bandwidth, loading 200 lines of import map adds about 7.5ms overhead in native Chrome, and about 10ms page load time in Firefox with ES Module Shims polyfill —— the cost of larger import maps is mostly low and as expected.

Project Future

ES Module Shims will continue to track upcoming new module features, with its baseline support target naturally changing over time.

• CSS Modules and JSON Modules are current new native module features, which move polyfill support baseline to opt-in through optional support, as they form static polyfill failures just like import maps.
• Supporting TLA is tricky, but when ensuring broad support is needed, it should be feasible by adding some lexer functionality.
• Unfortunately, module workers cannot be polyfilled in Firefox, as script workers never implemented dynamic import() (which, according to the first part, is needed to implement the loader). I believe this feature may come soon, if not already, then we'll have a polyfill path to inject loader hooks into module workers, possibly even polyfill features like module blocks on top of that.
• Due to Firefox never implementing import() for script workers, module workers cannot be polyfilled. I believe this feature may come soon, then we'll have a polyfill path to inject loader hooks into module workers, possibly even polyfill features like module blocks on top of that.
• Adding support for Wasm modules will also be a major goal of the project, possibly even combined with Wasm-related polyfills.
• It would also be interesting to try polyfilling asset references.

As a rule for the project, stable native features are only implemented when implemented in browsers. Otherwise, the polyfill mode risks diverging from native features.

Thank you for reading this far, now you have no reason not to use import maps! (And benefit from better fine-grained caching).

Contributors

XiSenao

Changelog

Last edited 2 months ago

View full history

9ba4ed7-feat: i18n

0240bc1-doc(material-strategy): How ES Module Shims became a Production Import Maps Polyfill