Repository navigation
Faster / parellel i18n AOT builds #6789
Description
Bug Report or Feature Request (mark with an x)
- [ ] bug report -> please search issues before submitting
- [ X ] feature request
Versions.
@angular/cli: 1.1.3
node: 6.10.1
os: darwin x64
@angular/animations: 4.2.4
@angular/common: 4.2.4
@angular/compiler: 4.2.4
@angular/compiler-cli: 4.2.4
@angular/core: 4.2.4
@angular/flex-layout: 2.0.0-beta.8
@angular/forms: 4.2.4
@angular/http: 4.2.4
@angular/material: 2.0.0-beta.7
@angular/platform-browser: 4.2.4
@angular/platform-browser-dynamic: 4.2.4
@angular/platform-server: 4.2.4
@angular/router: 4.2.4
@angular/cli: 1.1.3
Desired functionality.
I'm working on a fairly large app that uses AOT and i18n localization, and we need to translate into 60+ languages. Currently the only option is to run ng build for each language one at a time. For my app this takes 5 hours to complete running them in series. My short-term mitigation is to run ng build in parallel which will helps, but I think we can do better.
My thought is adding something like ng build -prod --i18n-config i18n-config.json
Then, in the config file you could list all the languages you want and the destination directories for each and ng-cli would run them in parallel. Additionally, it feels like it could do the AOT build once and hold the AST in memory then generate each language without having to re-do the entire AOT build for every language. I haven't fully researched this, and I know it'd add a lot of complexity, but for enterprise scale global apps this is absolutely needed to have reasonable build times.
Activity
chcaru commentedon Jun 23, 2017
I agree, this will enable the real world use case of having a large team with a large app that has continuous delivery and many languages. What is available right now simply doesn't scale.
filipesilva commentedon Jun 27, 2017
Yeah I can't really argue that 5h is a lot of time to build your app for all languages. I suppose at a low level, the apps array could be leveraged to that effect and used with a
ng build --allor something. Wouldn't be much different from just using a shell script like you do currently though.Saving part of the compilation from before sounds like a better idea, but much more complex. It would also require some explicit support from
ngcitself.intellix commentedon Jun 27, 2017
In the hopes of improving what we have, this is how we're currently compiling our site for each language. Let me know if you have anything better for parallelism :)
{ "scripts": { "build": "ng build --target=production -e ${NG_ENV:-prod} --aot", "build:i18n": "npm run build -- --output-path=dist/${LANG:-en} --bh /${LANG:-en}/ --i18n-file=src/locale/messages.${LANG:-en}.xlf --locale=${LANG:-en}", "heroku-postbuild": "for lang in en de sv; do LANG=$lang npm run build:i18n; if [ $? -ne 0 ]; then echo ERROR; exit 666; fi; done", "xi18n": "ng xi18n --i18n-format=xlf --out-file=locale/messages.xlf", } }Upon building in Heroku, it'll run the heroku-postbuild task which will start compiling languages one by one. If one of them fails then the whole build will fail. We originally tried to do things in parallel but an error would get swallowed up and report that the build was successful when it wasn't
ocombe commentedon Jul 18, 2017
We have the same problem at google and it is now reaching the limit of our build systems as well. We are working on improvements for v5, stay tuned!
intellix commentedon Nov 2, 2017
@ocombe just wanted to checkup on the timeline for the above stuff as v5 landed now :)
I'm guessing this is now a CLI thing since we have AOT watch mode and baking in the languages can be done much quicker?
ocombe commentedon Nov 2, 2017
It will not be parallel builds. We are working on something called "runtime i18n" right now, we'll bundle all translations in the same bundle and apply the translation at runtime (without needing a compiler, so this is still AOT-safe). This means only one build even if you have 100 languages.
It should land in 5.1 or 5.2 (depending if we find any blocker that requires more work).
SURENDARREDDY commentedon Nov 20, 2017
@ocombe , Anyway we are doing this for all the languages, Why can't we add for all environments.
I mean build only one for all languages and all environments once.
any chance?
aldo-roman commentedon Jan 19, 2018
@ocombe I think "runtime i18n" approach would allow the user to change the app language without losing app state. Currently, AFAIK, to change a language you have to reload the complete app again. You also need different bundles, which means more downloads.
Any clue on how this is going? Sounds really great.
chcaru commentedon Jan 19, 2018
@aldo-roman I think this is being tracked as part of this issue now:
angular/angular#16477
trakhimenok commentedon Feb 6, 2018
@ocombe is there any place we can read about reasons to have all translations in a single bundle? Why not to have a separate trans bundle per language?
ocombe commentedon Feb 6, 2018
we won't force you to have one bundle for all languages, you can build one bundle for each (it's what google will do internally)
9 remaining items
jordan112 commentedon Mar 1, 2019
Any updates on this? It was discussed coming in versions 5, then 6, then 7. 8 is around the corner. Our builds take approximately 45 minutes and we have about 20 languages. We will probably triple the number of languages and are looking at multiple hour builds. We are also discussing fanning out processing to many docker containers (one language per container) for every build, but wanted to check if there is any change in timeline so we can plan accordingly. Thanks!
sebastianzillessen commentedon Apr 3, 2019
Hi! I am looking as well forward to a solution for this.
Especially as when you are executing the build for multiple locales, you will have a lot of equal files (e.g.
runtime.**.js,styles.**.css,polyfill.**.jswill stay the same, only themain.**.jsfile will change according to the locale).Best,
Sebastian
MebWorker commentedon May 28, 2019
Is there a safe & tested solution for parallel build or speeding up languages builds now?
Wildhammer commentedon Jun 5, 2019
my solution for i18n parallel build:
https://stackoverflow.com/a/56463676/2582581
JanneHarju commentedon Aug 13, 2019
I'm trying to use https://www.npmjs.com/package/npm-run-all this package with --parallel flag. It works on my machine but will rise my CPU usage to 100% and in our CI machine it will throw memory allocation failed exception, but I think that it is only ram issue and we are testing if more RAM will fix it. Some say that you need to run ng command with extra flag with larger project like this:
"ng-high-memory": "node --max_old_space_size=8000 ./node_modules/@angular/cli/bin/ng"but yet it didn't fix my problem.dgp1130 commentedon Jun 26, 2020
Internationalization in Ivy is a lot smarter about builds so we don't need to do a full rebuild for each language by simply using the
--localizeparameter. You can specify only specific languages if you want, or build all of them and re-use most of the computation. This should drastically improve build performance.https://angular.io/guide/i18n#build-from-the-command-line
If you are still having performance/memory problems when using the Ivy process, then please open a new issue where we can look into that specific problem more closely.
angular-automatic-lock-bot commentedon Jul 27, 2020
This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.
Read more about our automatic conversation locking policy.
This action has been performed automatically by a bot.