Faster builds/rebuilds for development and production ⚡️ #11587

I like this direction too @santiagopuentep. It's not exactly an easy path, but I think it might be the best one.
We would have to change our way to load modules on the client-side, which would also change our way to do dynamic imports.

Anyway, I think the first task for this would be to list all the points we will have to change and post them in public so we can discuss alternatives.

Thanks for your feedback here

Yeah, without a change like this one, I don't think Meteor can ever become a big scale thing. Large projects are too slow, and this will get worse in the future IMO. Our project for example would have to be moved to some other system if the build times keep getting worse and worse. It's a shame because Meteor is an amazing platform otherwise.

With a lightning fast build system and moving everything to independent npm modules, I think this could be a future proof platform.

isobuild has long been a differentiator for Meteor. But I think the progress of ecsmascript combined with solutions like Vite have eroded that advantage.

I am under the impression that isobuild has required a lot of attention over the last 3-4 years. Being able to leverage another tool may be a big productivity win.

But... I know the cost to transition would likely be high.

Tough spot.

In Windows here Meteor is almost unusable. I'm using an Core i3 right now, but with 16 GB of RAM. Everything works just fine, like Storybook for example. I run Meteor and I get 100% CPU usage. In faster computers we get 30% or 40% usage, just by running Meteor. On mac the reload times very fast, in Windows they take forever.

The platform that doesn't strive to be the best, doesn't survive long term in the JS world. Having a good enough bundler is not enough, Vite doesn't even bundle in development. How can you compete with that? Meteor looks like an old dinosaur already, I love Meteor, but it's true. If it continues like this it won't survive a new technology doing something similar, but with Vite, full NPM support, etc.

@paulincai thanks for some tips.

We decided to split our large app (500k LOC) into microfrontoends(MC) and build it with ESBuild.

In two words:
We have a core(meteor app), which contain all common code like ui library, state manager and some business logic + meteor packages.

Every MC is npm package.

It is building with ESBuild. The config for ESBuild looks like:

import { exec } from 'child_process';
import path from 'path';
import { build } from 'esbuild';
import { lessLoader } from 'esbuild-plugin-less'; // as example, we use LESS for styles.

// isProduction flag for watch mode
const isProduction = process.env.NODE_ENV === 'production';
const MODULE_NAME = process.env.npm_package_name ?? '';
const NAME_PARTS = MODULE_NAME.split('/');
const isPrefixed = NAME_PARTS.length > 1;

const SAFE_NAME = NAME_PARTS.join('_'); // convert @company/package1 -> @company_package1

const DEV_LIVE_MODULES_PATH = path.resolve(`../../../public/dev-live-modules/`); //adjust it to your project structure
const OUTPUT_FILE_PATH = path.resolve(`./dist/${SAFE_NAME}.js`);

const copyToPublic = () => exec(`cd dist && npx copyfiles ./* ${DEV_LIVE_MODULES_PATH}`);

(async () => {
  build({
    watch: isProduction
      ? false
      : {
          onRebuild(error) {
            if (!error) {
              console.log('ReBuild succeeded');
              copyToPublic();
            }
          },
        },
    entryPoints: [path.resolve('./src/index.ts')], //entry point
    outbase: 'src',
    bundle: true,
    platform: 'node',
    format: 'cjs', // we should build it as Common JS module
    plugins: [lessLoader({})], // use any plugin you need
    // 1. mark meteor's packages as external
    // 2. some common packages also can be external (ui library etc)
    // 3. our "core" is located in /imports/ folder, ignore it too.
    external: ['mobx', 'react', 'react-dom', 'mobx-react', 'classnames', '@itgenio/*', 'meteor/*', '/imports/*'],
    outfile: OUTPUT_FILE_PATH,
    // banner and footer is for meteor "magic" - meteorInstall
    // generate it like meteor tools do it itself.
    banner: {
      js: `meteorInstall({'node_modules':{${
        isPrefixed ? `'${NAME_PARTS[0]}':{'${NAME_PARTS[1]}'` : '${MODULE_NAME}'
      }(require,exports,module) {`,
    },
    footer: {
      js: `${isPrefixed ? '}' : ''}}}},{extensions:['.js','.jsx','.ts','.tsx','.json','.less','.css','.html','.mjs']})`,
    },
  })
    .then(() => copyToPublic())
    .catch(e => console.error(e.message));
})();

In production, we use https://github.com/itgenio/meteor-live-modules (like original dynamic import) or add script to html (like @paulincai said)

In dev, we copy-paste it to public folder and meteor restarts with new code.

All you need after delivery your package into app - call require("@company/package1")

Pros:

• the build speed is super fast
• modular/isolated code
• independent release cycle for MC

Cons:

• you can't bundle meteor's package into MC (it can be solved with some plugin for esbuild)
• you should manually download/install MC (meteor's dynamic imports "just works")
• isolation of MC brings a new issue with communication between MC(but some good practice live "Event Bus" can solve it)

35 minutes is unusually slow. Could you please try building with the METEOR_PROFILE=100 env var set, and share the output here?

It actually built in 9 minutes on my M1. Taking 26 minutes and 21 seconds to build the Docker image. We're hosting our Meteor application on Kubernetes.

Not sure if you'd want to see the Dockerfile?

ESBuild doesn't support emitDecoratorMetadata option for TypeScript, but enabled in Meteor by default.
Another thing nested import and dynamic import which work because of reify implementation (AFAIK).

Can a plugin like this help: https://github.com/thomaschaaf/esbuild-plugin-tsc ?

For the reify imports, seems something to be raised with akyrum

Are you using Windows or Mac/Linux?

Linux (Ubuntu).

I have tweaked my build process for development to optimize my rebuilds. Overall it is working really well for me.

In addition to the tip above...

• to test try a meteor run with absolutely nothing else running, then compare to when you have editors open, etc
• on Windows make sure you disable virus scanning for your output folders
• Take a look at all process CPU usage to see if another process spikes while starting up. It would not be surprising to find other processes that are watching your output folder. (VSCode really goes nuts if you do not disable watching build folders)
• For most development I only need to build the modern build, use the --exclude-archs option, e.g. --exclude-archs=web.browser.legacy,web.cordova.
• For development you can also meteor run with the minimum platforms, e.g. only web.browser.
• Make sure you do not perform a meteor reset or rm -r .meteor/local/build between builds. Those commands will clear the build cache. Which can save you a lot of time.

I have used WSL for all development for the last 3-4 years and the build times are very manageable. My machine is a 15 yr old i7 running Windows 10 w/WSL.

Would love to see what Vite is capable of.

This is a good idea. Modern bundlers usually have an auto-eviction mechanism, as you mentioned. In the Meteor-Rspack integration, the cache managed by Rspack supports the maxAge setting: https://rspack.rs/config/cache#maxage.

maxAge (Default: 7 * 24 * 60 * 60 (7 days))
The maximum time, in seconds, that a cache version can remain unused. During cleanup, Rspack removes cache versions that have not been accessed for longer than maxAge. Set maxAge to Infinity to disable age-based cleanup.

However, the builder-cache folder you mentioned is managed by Meteor's bundler, so it would be good to have a specific solution on the Meteor side. I could imagine exposing this setting through the meteor configuration in package.json.

If anyone is willing to provide a solution like this in a PR, I'd be glad to review it and include it in a release. Otherwise, we will keep this suggestion on our list and try to implement it ourselves once we have more time after the current focus and developments.