Support top-level await in Node.js

Support for top-level await was just made available by default in Node v14.8.0 (nodejs/node#34558). We should enable this to be used in Pulumi programs, and look into what we need to do in terms of Node versions, flags for older Node versions, TypeScript versions, TypeScript compilation targets, and module system configuration to be able to (a) allow users to use top-level await (b) make this the default mode for new projects and update existing examples to take advantage of this where possible.

I saw TypeScript 4 just came out and was eager to see if I could get this working quickly. I think I got close, but alas, I ran out of time. I'm documenting what I did here in case it helps when we pick this up (which is hopefully soon!)

Manually compile your Pulumi program as an ES module with TypeScript 4:
- Run npm install -D typescript
- Add "main": "bin/" to package.json
- Add "type": "module" to package.json
- Change "target" in tsconfig.json to "es2017"
- Change "module" in tsconfig.json to "esnext"
- Disable automatic compilation with runtime:\n name: nodejs\n options:\n typescript: false in Pulumi.yaml
- Run ./node_modules/.bin/tsc
Patch the Pulumi program loader to use dynamic ES module loading, instead of require:
- Edit node_modules/@pulumi/pulumi/cmd/run/run.js
- Change (currently L214) from const reqResult = require(program) to const reqResult = import(program)

Alas, this doesn't work. The imported aws module is entirely empty in this program:

import * as aws from "@pulumi/aws";

const buckets = [];
for (const az of (await aws.getAvailabilityZones()).names) {
    buckets.push(new aws.s3.Bucket(`my-bucket-${az}`));
}

export const bucketNames = buckets.map(b => b.id);

This fails with a TypeError: aws.getAvailabilityZones is not a function, because it's undefined.

After digging more deeply, it looks like we'd need to recompile all of our packages as true ES modules and possibly rewire the way we re-export members. I tried things like

let getAvailabilityZones = import ("@pulumi/aws/getAvailabilityZones.js");
...
for (const az of (await ((await getAvailabilityZones) as any)()).names) {
...

to no avail. A lot of our module machinery seems to be done dynamically with requires and module.exports = ...s, and the ES module loader didn't like that. Maybe that's just TypeScript transpilation, but I have a sense we do a fair bit of this by hand. I have no idea if this can be changed seamlessly in a compatible way.

Along the way, I hit one particularly annoying issue, which appears to be nodejs/node#49441. Namely, many syntax errors (at least "unexpected reserved word") are hard to diagnose. You get errors like

    error: Running program '/Users/joeduffy/temp/ptsc' failed with an unhandled exception:
    SyntaxError: Unexpected reserved word
        at Loader.moduleStrategy (internal/modules/esm/translators.js:81:18)
        at link (internal/modules/esm/module_job.js:37:21)

when you ought to see something more like

file:///Users/joeduffy/temp/ptsc/bin/index.js:4
for (const az of (await aws.getAvailabilityZones()).names) {
                  ^^^^^

SyntaxError: Unexpected reserved word
    at Loader.moduleStrategy (internal/modules/esm/translators.js:81:18)
    at async link (internal/modules/esm/module_job.js:37:21)

This could harm debugging compile-time errors that TypeScript doesn't catch. In this case, I was simply using an older Node.js runtime that doesn't yet support top-level await, but it was a real PITA to track down.

After digging more deeply, it looks like we'd need to recompile all of our packages as true ES modules and possibly rewire the way we re-export members

This is the part I'm most worried about - that there is possibly (likely?) to be no way to support both non-ESM users and ESM users at once. I am not sure where the Node.js+TypeScript ecosystem is on this currently - but it feels like this point is at the crux of being able to move forward on this.

I'm a little confused about what we do in our modules. The @pulumi/aws/index.js file seems to try to "fake" ES modules?

"use strict";
// *** WARNING: this file was generated by the Pulumi Terraform Bridge (tfgen) Tool. ***
// *** Do not edit by hand unless you're certain you know what you are doing! ***
function __export(m) {
    for (var p in m) if (!exports.hasOwnProperty(p)) exports[p] = m[p];
}
Object.defineProperty(exports, "__esModule", { value: true });
__export(require("./getAmi"));
__export(require("./getAmiIds"));
...
const workspaces = require("./workspaces");
exports.workspaces = workspaces;
const xray = require("./xray");
exports.xray = xray;
//# sourceMappingURL=index.js.map

Notably the __esModule = true bit. Maybe this is standard TypeScript stuff?

Notably the __esModule = true bit. Maybe this is standard TypeScript stuff?

My understanding is that that was one of the hacks that Babel (and then TypeScript) adopted to make it so that ESM syntax could be used prior to ESM modules actually being supported in the target runtimes. So it allows import foo from "foo" even when the runtime doesn't actually support that. There are so many levels of these hacks over the years to paper over the fact that ESM wasn't really supported - and still is only sort of supported in Node via some forked path in the road that the vast majority of the ecosystem hasn't taken.

So I am confident that particular line of code does not suggest any actual support for truly loading these with current Node.js ESM support. That said, I don't know if there is now some target mode for TypeScript that lets things be emitted in such a way that they can be supported by both ESM and non-ESM module loading by consumers. My suspicion is that the answer is "no" - but I have found it surprisingly hard to find clear guidance on this in the ecosystem.

It's also possible that TypeScript has some answer purely at the TypeScript level that allows use of top-level await somehow without emitting true ESM modules. But I again suspect the answer is "no" here as well as top-level await really does depend on the runtime being in on it.

Someone will have to go deep on the current state of the ecosystem here to come up with a proposal for moving forward.

As an aside, Node 14 still issues this error when using ES modules, so there certainly be ample dragons there:

(node:6469) ExperimentalWarning: The ESM module loader is experimental.

Starting to port my company onto Pulumi and the combination of 2.0 forcing Promise returns and the crosswalk files being out of date creates friction.

I didn't think it would be a big deal to fix this myself but was surprised, even after updating my tsconfig.json:

    TSError: ⨯ Unable to compile TypeScript:
    index.ts(113,22): error TS1378: Top-level 'await' expressions are only allowed when the 'module' option is set to 'esnext' or 'system', and the 'target' option is set to 'es2017' or higher.

For people coming to this issue from google: just wrap promise returns in pulumi.output() or use async and indent.

Has there been any progress on this?

any updates?

[langhost/node] More ESM Support

This is now supported for JavaScript and pre-compiled TypeScript programs via support for ESM added in #7764. Some details on how to use this in #4130 (comment).

Additional support for features that enable this to be used via on-the-fly-compiled TypeScript programs (the most common use case in Pulumi) is coming in #8655.

[langhost/node] More ESM Support

[langhost/node] More ESM support: `nodeargs`, main entrypoints, and t…

This is now supported for on the fly compiled typescript.

How to:

update all your imports (https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c#file-esm-package-md)

Add to package.json:

{
    # ...
    "type": "module",
    "dependencies": {
        "ts-node": "^10.4.0",
        # ...
    }
}

Add to tsconfig.json:

    "target": "ESNext",
    "module": "ESNext",

Add to pulumi.yaml:

runtime:
  name: nodejs
  options:
    # See https://github.com/TypeStrong/ts-node/issues/1007
    nodeargs: "--loader ts-node/esm --no-warnings"

@awoimbee

Per @awoimbee's comment above, this is now as fully supported as it can currently be in Pulumi JS/TS. It is not yet available directly in the default Pulumi templates, and pulumi/templates#244 is tracking that. But given the dependence on experimental Node.js loader features, switching to using this by default will likely be something that is still a ways further off. In the meantime, it is easy for users to follow the approach in #5161 (comment) to start using ESM and top-level await today!

There is a working (and tested!) example of Top-level await in TS and Pulumi at https://github.com/pulumi/pulumi/tree/6bc7b1cc67621e515c44d9dde99ab8deba4d1c08/tests/integration/nodejs/esm-ts !

If you are having issues to make your setup work, try to compare with that code 👆 instead of spending as many hours as I have spent on debugging and googling this 😆

Changes for aws-native

for those coming across ERR_UNSUPPORTED_DIR_IMPORT error with imports should add --experimental-specifier-resolution=node to nodeargs like:

# Pulumi.yaml
runtime: 
  name: nodejs
  options:
    # See https://github.com/TypeStrong/ts-node/issues/1007
    nodeargs: "--loader ts-node/esm --no-warnings --experimental-specifier-resolution=node"

I got this sort of working with adding "type": "module" in package.json, settings in tsconfig.json and above in pulumi.yaml.

However as soon as I try to import a 3rd party package (e.g. ip-address) that isn't using "type": "module" I get the error: Cannot use import statement outside a module.

Anyone know a workaround for this?

@pjoe

I got this sort of working with adding "type": "module" in package.json, settings in tsconfig.json and above in pulumi.yaml.

However as soon as I try to import a 3rd party package (e.g. ip-address) that isn't using "type": "module" I get the error: Cannot use import statement outside a module.

Anyone know a workaround for this?

Hi @pjoe, you will need to find an ESM compatible version of ip-address or an alternative. Judging from
beaugunderson/ip-address#153 it seems ip-address is not yet available as ESM. One person says in the final comment that they started using https://github.com/sindresorhus/is-ip instead. I checked it is-ip with the pulumi ESM setup and it seems to work fine.

Similar to @zb-sj comment, you can specify the option in ts-config

TypeStrong/ts-node#422 (comment)

Navigation Menu

Repository navigation

Support top-level `await` in Node.js #5161

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Support top-level await in Node.js #5161

Description

Activity

joeduffy commented on Aug 21, 2020

lukehoban commented on Aug 21, 2020

joeduffy commented on Aug 21, 2020

lukehoban commented on Aug 21, 2020

joeduffy commented on Aug 21, 2020

keithharvey commented on Mar 12, 2021

suonto commented on Oct 15, 2021

thanhtoan1196 commented on Dec 11, 2021

lukehoban commented on Dec 30, 2021

awoimbee commented on Jan 11, 2022

lukehoban commented on Jan 17, 2022

zvictor commented on Mar 23, 2022

sejun-zigbang commented on Sep 29, 2022

pjoe commented on Jan 26, 2023

mkraenz commented on Feb 6, 2023

Sleepful commented on Apr 4, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Support top-level `await` in Node.js #5161