Dynamic imports at a CommonJS and ESM boundary

The original note used runtime dynamic imports to load ESM packages from an application with different compilation assumptions. It showed a shared loader, a libp2p factory, and length-prefixed stream helpers.

The historical loader

// To deal with ESM modules, we need to use dynamic imports
// Error: Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: No "exports" main defined in ...
const dynamicImport = async (packageName: string) => new Function(`return import('${packageName}')`)();

export async function initDynamicImports(libs: string[]) {
  return await Promise.all(libs.map((lib) => dynamicImport(lib)));
}

The helper puts import() inside a generated function, avoiding a transform of that expression by the surrounding compiler. It also interpolates a package name into executable source. Such input must remain trusted; this is not a loader for user-supplied strings. Native import() avoids that generated-source boundary when the build configuration preserves it appropriately.

ERR_PACKAGE_PATH_NOT_EXPORTED can indicate that the selected entry or subpath is not exposed by a package’s exports map. Dynamic import can select a different supported export condition, but it does not bypass the map or make an unexported path valid.

Load the libp2p components

import defaultsDeep from '@nodeutils/defaults-deep';
import { initDynamicImports } from '../libs/utils';
// import { createLibp2p as create } from 'libp2p';
// import { tcp } from '@libp2p/tcp';
// import { webSockets } from '@libp2p/websockets';
// import { noise } from '@chainsafe/libp2p-noise';
// import { tls } from '@libp2p/tls';
// import { yamux } from '@chainsafe/libp2p-yamux';
// import { gossipsub } from '@chainsafe/libp2p-gossipsub';
// import { bootstrap } from '@libp2p/bootstrap';
// import { mdns } from '@libp2p/mdns';
// import { SignaturePolicy } from '@libp2p/interface';
// import { identify } from '@libp2p/identify';

// https://docs.libp2p.io/guides/getting-started/javascript/
// https://github.com/libp2p/js-libp2p/blob/main/doc/CONFIGURATION.md
// https://github.com/libp2p/js-libp2p-examples/tree/main
export async function createLibp2p(_options) {
  const [
    libp2p,
    { tcp },
    { webSockets },
    { noise },
    { tls },
    { yamux },
    { gossipsub },
    { bootstrap },
    { mdns },
    { SignaturePolicy },
    { identify },
  ] = await initDynamicImports([
    'libp2p',
    '@libp2p/tcp',
    '@libp2p/websockets',
    '@chainsafe/libp2p-noise',
    '@libp2p/tls',
    '@chainsafe/libp2p-yamux',
    '@chainsafe/libp2p-gossipsub',
    '@libp2p/bootstrap',
    '@libp2p/mdns',
    '@libp2p/interface',
    '@libp2p/identify',
  ]);

  const defaults = {
    transports: [tcp(), webSockets()],
    streamMuxers: [yamux()],
    connectionEncrypters: [noise(), tls()],
    peerDiscovery: [mdns()],
  };
  return libp2p.createLibp2p(defaultsDeep(_options, defaults));
}

The arrays depend on matching package names and destructured results by position. The source does not pin the participating libp2p package versions, so current factory options and exports cannot be inferred from this example.

Initialize stream helpers before use

import { initDynamicImports } from '../libs/utils';
// import * as lp from 'it-length-prefixed';
// import map from 'it-map';
// import { pipe } from 'it-pipe';
// import { fromString as uint8ArrayFromString } from 'uint8arrays/from-string';
// import { toString as uint8ArrayToString } from 'uint8arrays/to-string';

let lp, map: any, pipe, uint8ArrayFromString, uint8ArrayToString;
const initLibs = async () => {
  if (!lp) {
    [lp, map, { pipe }, uint8ArrayFromString, uint8ArrayToString] = await initDynamicImports([
      'it-length-prefixed',
      'it-map',
      'it-pipe',
      'uint8arrays/from-string',
      'uint8arrays/to-string',
    ]);
    map = map.default;
    uint8ArrayFromString = uint8ArrayFromString.fromString;
    uint8ArrayToString = uint8ArrayToString.toString;
  }
};

export async function stdinToStream(stream) {
  await initLibs();
  // Read utf-8 from stdin
  process.stdin.setEncoding('utf8');
  pipe(
    // Read from stdin (the source)
    process.stdin,
    // Turn strings into buffers
    (source) => map(source, (string) => uint8ArrayFromString(string)),
    // Encode with length prefix (so receiving side knows how much data is coming)
    (source) => lp.encode(source),
    // Write to the stream (the sink)
    stream.sink,
  );
}

export async function streamToConsole(stream) {
  await initLibs();
  pipe(
    // Read from the stream (the source)
    stream.source,
    // Decode length-prefixed data
    (source) => lp.decode(source),
    // Turn buffers into strings
    (source) => map(source, (buf) => uint8ArrayToString(buf.subarray())),
    // Sink function
    async function (source) {
      // For each chunk of data
      for await (const msg of source) {
        // Output the data as a utf8 string
        console.log('> ' + msg.toString().replace('\n', ''));
      }
    },
  );
}

The source lazily initializes module variables and launches stream pipelines. It does not await or return those pipeline results, so the exported async function finishing does not establish that the stream finished successfully. Concurrent initialization and stream/standard-input ownership also need a deliberate lifetime policy.

No modules, peer connections, or pipelines were executed here. References: Node package entry points and dynamic import.

← Back to the garden