Skip to content

Heapdump generation as part of Node core #279

Closed
Closed
Heapdump generation as part of Node core#279

Description

@mhdawson
mhdawson
opened on Mar 7, 2019

Was discussed in Diagnostic summit today. Consensus was that ability to generate heapdumps should be in core versus needing modules like Heapdump.

We should work on

  • Pulling sampling heap profiler into core
  • Offering ability to generate heapdump as part of core APIs and on events like Unhandled Exception, etc. through command line options.

Lets use this issue to flesh out the APIs' functionality that should be exposed.

Activity

  1. mhdawson commented on Mar 7, 2019

    @mhdawson
    mhdawson
    on Mar 7, 2019
    MemberAuthor

    For references, older issue with some past discussions on the topic: nodejs/TSC#257

  2. sam-github commented on Mar 7, 2019

    @sam-github
    sam-github
    on Mar 7, 2019

    Note that the core capability is already present:

    const fs = require('fs');
const inspector = require('inspector');
const session = new inspector.Session();

session.connect();

fs.writeFileSync('_.heapsnapshot', '');

session.on('HeapProfiler.addHeapSnapshotChunk', function(m) {
  fs.appendFileSync('_.heapsnapshot', m.params.chunk);
});

session.post('HeapProfiler.takeHeapSnapshot', null, function(err, r) {
  console.log('Runtime.takeHeapSnapshot done:', err, r)
  session.disconnect();
});

    It probably needs an example usage, like: https://nodejs.org/api/inspector.html#inspector_cpu_profiler

  3. jasnell commented on Mar 7, 2019

    @jasnell
    jasnell
    on Mar 7, 2019
    Member

    I'll be opening a pr that essentially pulls in node-heapdump ... At least most of it. Expect that soonish

  4. misterdjules commented on Mar 7, 2019

    @misterdjules
    misterdjules
    on Mar 7, 2019

    @jasnell My experience with node-heapdump has been that very often, it's not usable because it requires the process to be able to allocate as much memory as what is used for the JS heap, and that leads to the process actually crashing. Has there been further discussions during the summit about solving that problem?

    I know for instance that there's been discussions in the past about being able to generate heapdumps from core dumps (e.g. in nodejs/post-mortem#13).

    I'm not suggesting that this should block moving node-heapdump into node's core, but I'm wondering whether we should also think about a different approach that doesn't have node-heapdump's issues.

    That being said, I'm also interested in hearing about the experience that others had with node-heapdump, there may be something that I'm missing that would solve the issues that I've experienced.

  5. jasnell commented on Mar 7, 2019

    @jasnell
    jasnell
    on Mar 7, 2019 via email
    Member

  6. misterdjules commented on Mar 7, 2019

    @misterdjules
    misterdjules
    on Mar 7, 2019

    @jasnell

    There will be parallel efforts to see what is possible around using coredump.

    Also just wanted to make it clear that I'm not saying generating heap dumps from core dumps is necessarily what I'm looking for. I'm looking for a robust/reliable way to generate data that gives insights into the content of the JS heap, whether this is done via core dumps or another mechanism is an implementation detail for me.

  7. joyeecheung commented on Mar 7, 2019

    @joyeecheung
    joyeecheung
    on Mar 7, 2019
    Member

    Previous discussions stalled on the format in which the heap dump should be returned to the user land: nodejs/node#23328

    Note that we already have internalBinding('heap_utils').createHeapDump() that returns the heap snapshot as a JSON.parsed object. So if we are exposing this functionality to the users I'd expect we work on top of that instead of pulling in node-heapdump, which mostly just adds some convenient signal handlers and a file writer for that.

  8. naugtur commented on Mar 7, 2019

    @naugtur
    naugtur
    on Mar 7, 2019
    Contributor

    The convenience of using signals by just adding a --require heapdump when running existing code without modification was nice though. Being able to do that with a flag (because someone might be using signals for something else I suppose) would be a nice equivalent.
    If it doesn't warrant adding a flag, it makes sense to keep a simplistic heapdump module out there to enable the functionality.

  9. alexkozy commented on May 16, 2019

    @alexkozy
    alexkozy
    on May 16, 2019
    Member

    I'd like to mention CLI tools that I built over last weekends. It utilizes inspector protocol to capture different profiles and tracing for given node process, it supports child processes, ondemand profiling and will support workers as well soon. It can capture sampling heap profiler.
    Please take a look: https://github.com/ak239/thetool . It is npm installable module with only one dependency (update-notifier).

  10. vmarchaud commented on May 28, 2019

    @vmarchaud
    vmarchaud
    on May 28, 2019
    Contributor

    I believe this can be closed since both the heap sampling profiler and the heapsnapshot can be launched with Node core.

  11. mhdawson commented on May 28, 2019

    @mhdawson
    mhdawson
    on May 28, 2019
    MemberAuthor

    @vmarchaud would you happen to have the link to the PR for adding the sampling profiler? It would be good to have that in this issue and then I think I can close.

  12. vmarchaud commented on May 28, 2019

    @vmarchaud
    vmarchaud
    on May 28, 2019
    Contributor

    @mhdawson Here is it : nodejs/node#27596

  13. mhdawson commented on May 28, 2019

    @mhdawson
    mhdawson
    on May 28, 2019
    MemberAuthor

    @vmarchaud thanks !

  14. mhdawson
    closed this as completedon May 28, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions