Skip to content

Wekan Crash if more than 2GB of RAM is used #3585

Closed
Closed
Wekan Crash if more than 2GB of RAM is used#3585
Labels
Bug

Description

@r4nc0r
r4nc0r
opened on Feb 18, 2021

Issue

Other Wekan issues can be added here.

Server Setup Information:

  • Did you test in newest Wekan?: yes
  • For new Wekan install, did you configure root-url correctly so Wekan cards open correctly https://github.com/wekan/wekan/wiki/Settings ?
  • Wekan version: 4.96
  • Deployment Method: docker swarm
  • Http frontend if any (Caddy, Nginx, Apache, see config examples from Wekan GitHub wiki first):
  • Node Version:
  • MongoDB Version: 4.0.11

Problem description:
if NodeJs uses more than 2GB of Memory, Wekan crashes with a Exception:

I researched a little and as it turns out NodeJs v12 has a standard limit of 2GB Memory Consumption, but it can be changed with the following Nodejs CLI Command: --max-old-space-size xxxx where xxxx is the memory in MB (source) Can this be implemented to raise the Wekan max Memory Limit?

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory,
,
<--- Last few GCs --->,
,
[1:0x38a2e80] 324979 ms: Mark-sweep 2029.2 (2080.6) -> 2018.8 (2083.1) MB, 1608.6 / 0.1 ms (average mu = 0.120, current mu = 0.022) allocation failure scavenge might not succeed,
[1:0x38a2e80] 327768 ms: Mark-sweep 2031.4 (2083.1) -> 2022.9 (2063.4) MB, 2738.9 / 0.1 ms (+ 0.1 ms in 30 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 2751 ms) (average mu = 0.061, current mu = 0.018) allocation f,
,
<--- JS stacktrace --->,
,
==== JS stack trace =========================================,
,
0: ExitFrame [pc: 0x140a859],
Security context: 0x22ae44d408d1 ,
1: changeField [0x369877f3a6b9] [/build/programs/server/packages/ddp-server.js:~287] [pc=0x273d069851f8](this=0x36d784d60269 ,0x06e82165b7a9 <String[18]: Ngb63ADH8RE3BT3C8Y>,0x24027c246289 <String[#6]: status>,0x1a40249114f9 <String[6]: online>,0x36d784d603e1 ,0x24027c240639 ),
2: ...,
,
1: 0xa18150 node::Abort() [node],
2: 0xa1855c node::OnFatalError(char const*, char const*) [node],
3: 0xb970be v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node],
4: 0xb97439 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node],
5: 0xd546b5 [node],
6: 0xd54d46 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node],
7: 0xd61605 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node],
8: 0xd624b5 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node],
9: 0xd64f6c v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node],
10: 0xd2b98b v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node],
11: 0x106e0bc v8::internal::Runtime_AllocateInOldGeneration(int, unsigned long*, v8::internal::Isolate*) [node],
12: 0x140a859 [node],

Activity

  1. xet7
    added
    Bug
    on Feb 19, 2021

  2. xet7 commented on Feb 19, 2021

    @xet7
    xet7
    on Feb 19, 2021
    Member

    Related https://github.com/wekan/wekan/discussions/3580

  3. r4nc0r commented on Feb 19, 2021

    @r4nc0r
    r4nc0r
    on Feb 19, 2021
    Author

    Do you mean the env var NODE_OPTIONS: --max_old_space_size=4096? If this is enough to fix the problem it would be great :)

  4. xet7 commented on Feb 19, 2021

    @xet7
    xet7
    on Feb 19, 2021
    Member

    @r4nc0r

    Please try does it work.

  5. r4nc0r commented on Feb 22, 2021

    @r4nc0r
    r4nc0r
    on Feb 22, 2021
    Author

    Works :)

  6. r4nc0r commented on Feb 22, 2021

    @r4nc0r
    r4nc0r
    on Feb 22, 2021
    Author

    Could you add this to the offical wekan documentation?

  7. r4nc0r
    closed this as completedon Jun 7, 2021
  8. xet7
    reopened this on Jul 3, 2021

  9. jankapunkt commented on Jul 3, 2021

    @jankapunkt
    jankapunkt
    on Jul 3, 2021
    Contributor

    The real question is why it's using >2GB of RAM.

    Are there any unbusted chaches?
    Memory leaks?
    Too many publictations/observers?
    Assets Like Images loaded into RAM?

  10. xet7 commented on Jul 3, 2021

    @xet7
    xet7
    on Jul 3, 2021
    Member

    @jankapunkt

    Yes sure. When Wekan is running, CPU usage and RAM usage is increasing all the time, as mentioned in some of these issues.

  11. xet7 commented on Jul 3, 2021

    @xet7
    xet7
    on Jul 3, 2021
    Member

    Also too much data is published to browserside, making browsers crash #3893

  12. Ben0it-T commented on Nov 5, 2021

    @Ben0it-T
    Ben0it-T
    on Nov 5, 2021
    Contributor

    Definitely a stupid question ...
    Should we add the env var NODE_OPTIONS: --max_old_space_size=4096 with snap (for prevent Wekan crashes if NodeJs uses more than 2GB of Memory) ?

    If yes, where/how ?
    On Debian 10, the unit file snap.wekan.wekan.service, we can see
    EnvironmentFile=-/etc/environment

    Should we add the env var NODE_OPTIONS: --max_old_space_size=4096 in /etc/environment ?

    Thanks

  13. xet7 commented on Nov 5, 2021

    @xet7
    xet7
    on Nov 5, 2021
    Member

    @Ben0it-T

    There are no stupid questions. There are only my stupid answers.

    WeKan Snap does not have that setting option yet.

    To try that option, you could use start-wekan.sh from https://github.com/wekan/wekan or some service file, and bundle version from https://github.com/wekan/wekan/wiki/Raspberry-Pi using bundle version for your server CPU architecture.

    If adding that option helps, that setting could be added to some Snap settings at https://github.com/wekan/wekan/tree/master/snap-src/bin .

    If that option does not help, then you should wait when I have finished partial stupid rewrite of WeKan.

  14. Ben0it-T commented on Nov 6, 2021

    @Ben0it-T
    Ben0it-T
    on Nov 6, 2021
    Contributor

    @xet7
    Thank you for your answer

    For the moment I have not yet deployed Wekan within our team. I use it for my personal use.
    I plan to deploy it in a few weeks for our team. So I have not yet encountered this type of problem...

    At first, it could probably be interesting to know the details about the heap and size, with getHeapStatistics().
    Maybe in Admin Panel > Version ?

    • total_heap_size
    • total_heap_size_executable
    • total_physical_size
    • total_available_size
    • used_heap_size
    • heap_size_limit
    • ...

    What do you think about that ?

  15. xet7 commented on Nov 6, 2021

    @xet7
    xet7
    on Nov 6, 2021
    Member

    @Ben0it-T

    You can send PR to add them to:

    Thanks!

  16. Ben0it-T commented on Nov 12, 2021

    @Ben0it-T
    Ben0it-T
    on Nov 12, 2021
    Contributor

    @xet7
    A little feedback on my stupid question about NODE_OPTIONS with snap
    On the latest Wekan version (5.75) with snap, if I put NODE_OPTIONS in /etc/environment, it works !

    Ex1 :

    root@xxxxxxxx:~# cat /etc/environment
# Nodejs
NODE_OPTIONS="--max_old_space_size=2048"
root@xxxxxxxx:~# snap restart wekan

    max_old_space_size_2048

    Ex2 :

    root@xxxxxxxx:~# cat /etc/environment
# Nodejs
NODE_OPTIONS="--max_old_space_size=4096"
root@xxxxxxxx:~# snap restart wekan

    max_old_space_size_4096

  17. xet7 commented on Jan 19, 2022

    @xet7
    xet7
    on Jan 19, 2022
    Member

    Somebody got this:

    FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

<--- Last few GCs --->

[1:0x2d9ee80]  1270397 ms: Mark-sweep 2036.4 (2065.4) -> 2030.9 (2066.9) MB, 1982.8 / 0.2 ms  (average mu = 0.185, current mu = 0.131) allocation failure scavenge might not succeed
[1:0x2d9ee80]  1272372 ms: Mark-sweep 2036.5 (2066.9) -> 2032.0 (2067.9) MB, 1951.8 / 0.2 ms  (average mu = 0.098, current mu = 0.012) allocation failure GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x140dff9]
Security context: 0x1f42ab6808d1 <JSObject>
    1: changeField [0x3899d570e079] [/build/programs/server/packages/ddp-server.js:~287] [pc=0x158bfe4d16b8](this=0x09e2b6008079 <SessionDocumentView map = 0xff51671a929>,0x00191a67ef61 <String[18]: NdHSHs6pNEj8woP53M>,0x2babc7c8c171 <String[#4]: sort>,0x3fc3dcd2d069 <HeapNumber -2>,0x09e2b60081f1 <Object map = 0x292df8f7e0a9>,0x1a2b41b80639 <true>)
    2: added(...

 1: 0xa1a640 node::Abort() [node]
 2: 0xa1aa4c node::OnFatalError(char const*, char const*) [node]
 3: 0xb9a9fe v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xb9ad79 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xd57ff5  [node]
 6: 0xd58686 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
 7: 0xd64f45 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xd65df5 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xd688ac v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0xd2f2cb v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
11: 0x10719fc v8::internal::Runtime_AllocateInOldGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
12: 0x140dff9  [node]

    So does any of these help?

    https://stackoverflow.com/questions/53230823/fatal-error-ineffective-mark-compacts-near-heap-limit-allocation-failed-javas

    @mfilser

    Is there already some setting in WeKan for this, or should it be added?

  18. xet7
    pinned this issue on Jan 19, 2022

  19. xet7 commented on Jan 25, 2022

    @xet7
    xet7
    on Jan 25, 2022
    Member

    Some progress:

    I did this testing on Debian VM that is running on bare metal server donated to WeKan development. That Debian VM has 60 GB RAM, just in case, it could work with less.

    1. So that right click Inspect console would not show Templates related errors, I clicked right top username / Templates, to open Templates board and have those created.
    2. One customer gave me example board export, size 4 MB of json.
    3. I checked node path:
    which node
    1. In start-wekan.sh I have added these, with node path, and additional stack size mentioned here https://stackoverflow.com/questions/20936486/node-js-maximum-call-stack-size-exceeded

    UPDATE: This works best with size 4096. Bigger values do increase CPU usage too much.

    cd .build/bundle
export NODE_OPTIONS="--max_old_space_size=4096"
/bin/bash -c "ulimit -s 65500; exec /home/wekan/.volta/bin/node --stack-size=65500 main.js"
    1. At Debian VM, I installed xclip, that copies textfile to clipboard:
    sudo apt -y install xclip
    1. I copied board JSON to clipboard:
    cat board.json | xclip -se c
    1. In Chrome, I did: All Boards page / Add Board / Import / From previous export / right click to input field / paste
    2. (I did not map any users in my case, because my test WeKan install has only one user. This is optional.)
    3. There could be another button about mapping users, without list of users to map. In this step, I checked is there any errors at browser right click Inspect Console. There were no errors. Anyway, I did not click again (sometime it could cause duplicate imported boards), I just clicked All Boards, and opened that imported board.
    4. Memory usage in that Debian VM:
    $ free -h
               total        used        free      shared  buff/cache   available
Mem:            61Gi       3,4Gi        47Gi       135Mi        10Gi        57Gi
Swap:          975Mi          0B       975Mi
    1. I don't see any errors in browser right click Inspect console, other than one notice about autofocus that does not matter.
  20. xet7
    added a commit that references this issue on Feb 3, 2022

    Add more stack and heap to Node.js at all WeKan platforms.

    ff13571
  21. xet7
    closed this as completedon Feb 4, 2022
  22. xet7
    unpinned this issue on Feb 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Bug

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions