path to restoring FreeBSD to Tier 2 support

What would have to happen to restore FreeBSD to Tier 2 support? Here's the tiniest of beginnings of a task list. Feel free to add to it. @nodejs/build

Need a volunteer to do sys admin, troubleshooting, documentation, ansible-scripting, etc. for FreeBSD in CI. (If that person is an employee of a company invested in Node.js/FreeBSD, that would be a big plus, but not strictly a requirement.)
... what else?

Refs: nodejs/node#27383

There is a public FreeBSD CI service now too https://forums.freebsd.org/threads/cirrus-ci-support-for-freebsd.68644/

Do we need to acquire one or more people to be the go-to FreeBSD experts when it comes time to create a release? I imagine that would be the Build WG member or members. I assume we don't need someone on the Release WG for that. But typing this out loud in case I'm wrong.

There is a public FreeBSD CI service now too https://forums.freebsd.org/threads/cirrus-ci-support-for-freebsd.68644/

Thanks! I do think we're going to continue to need FreeBSD in our existing CI. Public CI is not an option in a lot of situations for us, such as when we are doing security releases. And even if it were, I'm not sure fragmenting our CI across multiple services is a path to less maintenance pain instead of more (but I could be wrong!).

Putting on the meeting agenda even though I only opened it 15 minutes ago because:

I believe this is worth talking about very publicly.
This does not seem like the type of thing that warrants weeks of open-ended discussion. I believe if we had a 10-minute synchronous conversation, we could probably have the checklist completed. So let's just do that.

IIUC the idea behind the downgrade is to "officially-unofficially" move freeBSD to the community-contrib path (#1788. Similar to Linux/ia32 and ARM6)

Need a volunteer to do sys admin...

I believe the other part is maintaining a robust @nodejs/platform-freebsd team to own the code side...

Need a volunteer to do sys admin...

I believe the other part is maintaining a robust @nodejs/platform-freebsd team to own the code side...

Yes! I think this comment explains a bit what our problem is regarding FreeBSD support: nodejs/node#27383 (comment)

If FreeBSD is a release-blocking platform, we have to fix any build issue in nodejs/node. It happened several times that a V8 upgrade was blocked only because of FreeBSD and we spent more time than necessary to fix the issues because no expert was here to help. Since issues had to be fixed by us in the first place, it might be why the FreeBSD port "just works".

If FreeBSD is a release-blocking platform, we have to fix any build issue in nodejs/node. It happened several times that a V8 upgrade was blocked only because of FreeBSD and we spent more time than necessary to fix the issues because no expert was here to help. Since issues had to be fixed by us in the first place, it might be why the FreeBSD port "just works".

it sounds like a lot of this is due to lack of communication between teams. aside from being sure FreeBSD developers are subscribed to this and the main nodejs issue feeds, are there other less noisy ways to ensure problems like you mention do not get missed? personally speaking i'm sub'd to nodejs/build and nodejs/node projects on github (with some filtering to lower the signal/noise ratio), but are there other communication channels FreeBSD should also be keeping an eye on?

but are there other communication channels FreeBSD should also be keeping an eye on?

We have the @nodejs/platform-freebsd team, but for $GITHUB_REASONS you need to be a members of the Org to subscribe to that team's notifications.

I'd personally be OK with adding FreeBSD members as nodejs collaborators even if they don't have a nodejs contributions history, if that's what it takes for them to be get notified of issues, and start being contributors.

@Trott

On the infra side: it would be good to have some FreeBSD natives who are somewhat present around nodejs/build (casually following the repo and watching for ways to jump in) and in the nodejs/platform-freebsd team as well so we can use that to summon help where needed.

But we still have a problem with turn-around, @Trott this goes back to our discussion about urgency and downtime: there's one class of problem where there's just something that needs to be taken care of but is not super urgent (like FreeBSD machines not having an auto-restart mechanism for the Jenkins process which was a huge headache for the longest time until someone finally put monit on them). But then there's another class of problem where something's failing right now and someone has to fix it. That falls on those of us who are more present—who watch the repo, who are on IRC and who tend to get pinged when there are urgent matters (because we tend to be responsive: myself, Refael and Michael get that the most at the moment I think).

This second class is where the pain is felt and I don't think that can be solved by casual involvement. The reason FreeBSD was easy for us earlier on was because we had Johan who was a core Build member, was very active and available, he was also a FreeBSD native (which is why we still have things like the linter running on FreeBSD). Without a Johan around these urgent things fall on me and the other active members and this is where the biggest problem is.

Back to my comments on nodejs/node#27383: the experience for me of managing FreeBSD and picking up pieces when things fail (combined with a brief and disastrous run with FreeNAS at home), have made me really dislike FreeBSD. And it's my strong impression that its presence in our CI system is much more of negative than a positive for the work that we need to get done here. I feel almost exactly the same way about SmartOS to be honest but it gets more of a free pass because at least Joyent is a major infrastructure contributor (and Colin et. al. are usually responsive to calls via nodejs/platform-smartos).

Our infrastructure make-up is really driven very strongly by the individuals, and companies, that contribute here. It's not all about Node core needs, there's a big element of preferences. e.g. we have a ton of Ubuntu in our mix, it runs some our most important servers and it's the base OS for a lot of our Docker container work—that's almost entirely about my personal preferences, not necessarily technical merit. If Johan was still an active contributor then we'd probably see a much stronger presence of FreeBSD and Debian.

If you like FreeBSD, then there's a good chance you're a bit of a systems nerd and you might even enjoy the work we do around here across many systems. So maybe you should consider getting more involved and becoming one of those people that reacts to urgent needs. If we had that, and the core dev team didn't have the sense that FreeBSD gets in the way (as @targos pointed out above), then I'm sure it could easily raise in status.

As Rod said its all about having one or more people who are active enough in the project and build WG to champion FreeBSD and be responsive enough to make sure it is not a burden on others. This includes the full cycle from helping to keep the machines configured and up to date following the build WG standard practices (ansible templates etc.) to being responsive when there are machine issues or test failures on the platform.

@sam-github

@sam-github anyone can be in the platform-* teams. They don't give special permissions.

@anandsuresh

@anandsuresh I've sent you an invitation to join the platform-freebsd team. (I believe you won't be able to join unless you have two-factor authentication enabled on your GitHub account.)

Testing on my local machine running FreeBSD 12.0-RELEASE-p10, I was able to build and test the latest code in master without any issues. The output from the test run is available at https://gist.github.com/anandsuresh/e8a769e19d157718115a4f9dd6c8de6f.

It seems that the issue is specific to FreeBSD v10 or v11. I don't have any resources/jails with those specs at the moment. Will try and set something up to test further.

Ran the same test in a FreeBSD 11.3 jail and discovered that it failed with the following errors, but passed when I can the test command again. Is there a history of flaky tests in this suite?

=== release test-net-connect-local-error ===
Path: sequential/test-net-connect-local-error
assert.js:89
  throw new AssertionError(obj);
  ^

AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
+ actual - expected

+ undefined
- 12347
    at onError (/wrkdirs/usr/ports/www/node/work/node-v12.10.0/test/sequential/test-net-connect-local-error.js:26:10)
    at Socket.<anonymous> (/wrkdirs/usr/ports/www/node/work/node-v12.10.0/test/sequential/test-net-connect-local-error.js:36:49)
    at Socket.<anonymous> (/wrkdirs/usr/ports/www/node/work/node-v12.10.0/test/common/index.js:371:15)
    at Socket.emit (events.js:209:13)
    at emitErrorNT (internal/streams/destroy.js:91:8)
    at emitErrorAndCloseNT (internal/streams/destroy.js:59:3)
    at processTicksAndRejections (internal/process/task_queues.js:80:21) {
  generatedMessage: true,
  code: 'ERR_ASSERTION',
  actual: undefined,
  expected: 12347,
  operator: 'strictEqual'
}
Command: out/Release/node /wrkdirs/usr/ports/www/node/work/node-v12.10.0/test/sequential/test-net-connect-local-error.js
[02:57|% 100|+ 2742|-   1]: Done
gmake[1]: *** [Makefile:296: jstest] Error 1
gmake: *** [Makefile:330: test-only] Error 2

If it helps, the machine I'm running is a 2011 Mac Mini with 16GB of RAM and 1TB of storage that has been wiped clean to run FreeBSD 12.0-RELEASE-p10 on the host, with 2 poudriere jails used to build the node.js port on FreeBSD 12.0-RELEASE-p10 and 11.3-RELEASE-p3.

@anandsuresh

@anandsuresh at a guess it's IPv6 problems. On all our test machines we make sure that this line is in /etc/hosts, maybe try it:

::1 localhost.localdomain localhost

common.localhostIPv6 is ::1 so it needs to be present

A couple of points, bearing in mind that I am far from an expert in these matters:

The major problem may be related to FreeBSD 10, which has reached end-of-life. The version of clang that was shipped with FreeBSD 10 had (in my opinion) a number of problems. Those problems no longer exist on current release versions. Keeping the build going on currently-supported versions of FreeBSD may therefore be substantially easier than it was.
If the issue is a lack of hardware resources (i.e. servers to build/test/CI FreeBSD 11 and/or FreeBSD 12), my company can donate VM's to meet that need. (I see above that FreeBSD 12 may be an issue with your current providers.) Who would I contact about that?

FYI we now have no FreeBSD 10 running in CI, we just need to deprovision the VMs, #2012.

@jdwx

@jdwx it's just as much about having people who will tend/care for the machines, investigate machine issues and test failures. Would your company be able to donate some of your time to help on that front?

@mhdawson

@mhdawson:

"tend/care for the machines"

Yes.

"investigate machine issues"

Yes.

"test failures"

Probably not much. Our Node.JS experience is very limited and our collective knowledge of its internals amounts to basically nil.

The VM(s) we provide would be fully managed for you, but on the development side the best we could probably do is act as a resource for questions like, "WTF is FreeBSD actually doing here?" to the best of our ability; our knowledge of FreeBSD internals is substantial, but by no means exhaustive. But it sounds like maybe that's right about where the current gap is. So while it might not be as good as having all the relevant knowledge in one person's head, perhaps it would suffice?

Just for add on discuss, I will share the history about my experience as consultant:

1 year ago I have left some FreeBSD servers from my clients because they was willing test the node.js and Vue / Nuxt (last one because native SSR ready) or other Javascript frameworks.

When they asked me to enable the support on FreeBSD 10 (currently LTS on that time), it was full of problems, and then clients was claiming about hire other consultants (so much time trying fixing errors) to do the Job, in another words, they does not care about if node.js was runing on Linux / FreeBSD on first time for they purpose / test ...

Well, for do not lose my contracts, then I have used Ubuntu and archive they requests...

After sometime, the number of OSs about fix stuff on Linux increased, and since I charge per call, they quit about use the JavaScript frameworks claiming those as headache…

Usually (making a big generalism) there 2 type of FreeBSD clients:

1 - Those with simple requirements focusing on stability, and since they requirements are simple the default ports works pretty well

2 - Some with advanced requirements trying get the stability of FreeBSD for enhance they heterogeneous (using FreeBSD everywhere is possible, and where is not, they use another best alternative) environment.

1 and 2 know about how solid FreeBSD is, and this is what make all difference… Is not bias or personal preference...

For those thinking this way (as personal opinion) so I suggest you to start pay for every call for your outsource company where system's problems occurs with popular Linux versions (Enterprise like Red hat since you pay an fee you can use they support and is not the point here).

Is that true mostly FreeBSD clients at some point don't renew they contracts for phew time, but when they come back, mostly of time the outdate environment still solid as rock (Got 2 months ago an request for update an FreeBSD 8 mail server to 12 for example), in another words, this is the casual client and is backing saying about "long term contract" and blah blah blah, but the truth, only wish upgrade they environment...

From point of view on enterprises, systems requiring less (read spend less money) support calls are the best option ever...
Nowadays I am testing again FreeBSD for node and Nuxt, this time FreeBSD 12.0 compiles everything on first try.

By the way, using PM2 for keep restarting the process have a lot of issues, seems they are looking how to fix the needed about run as root (never will use it on real production environment with this kind of behavior – if apache I Jail it and never run as root independent of they reputation, why would let pm2 or any other JavaScript running as root changing the entire environment…)

Second problem, speaking strictly about Javascript, imho there’s a big problem about keep everything update in sync…
Get the previous example…

On the PM2 github, mostly of “workarounds” about the previous issue is about downgrade node.js for specific version where “all was working”…
This kind of comment is the first factor to drop JavaScript from projects on meetings in IT enterprises, no one wish become the responsible for start projects with outdate versions with lot of CVE’s (security issues) only for speed up the development because one or another framework…
I will not discuss the problem about the repositories… Need be crazy for run arbitrary scripts on production servers as root, this is why the requirement about jail or use unpowered per process is the basic concept as security point of view…

Remembering running inside jail as root can ruin all your job too… Or in another words, if you will spend mostly time fixing things instead be productive, this will never be an good option.

I hope all discuss here by other people helping make node.js run pretty well on FreeBSD can consider those points, the idea is let you guys think about from enterprise perspective. In another words, for increase the popularity about node.js, the main way is support all systems possible, because every enterprise have distinct requirements, and if node.js can’t support those, will never be an option.

This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.

@anandsuresh

@anandsuresh as far as this issue is concerned, if you're finding FreeBSD 12 works fine and 11 or earlier has issues, we should just focus our efforts there.

Navigation Menu

Repository navigation

path to restoring FreeBSD to Tier 2 support #1791

17 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

path to restoring FreeBSD to Tier 2 support #1791

Description

Activity

Trott commented on Apr 25, 2019

nschonni commented on Apr 25, 2019

Trott commented on Apr 25, 2019

Trott commented on Apr 25, 2019

Trott commented on Apr 25, 2019

refack commented on Apr 25, 2019

refack commented on Apr 25, 2019

targos commented on Apr 25, 2019

nomadlogic commented on Apr 25, 2019

refack commented on Apr 25, 2019

sam-github commented on Apr 25, 2019

rvagg commented on Apr 26, 2019

mhdawson commented on Apr 26, 2019

17 remaining items

targos commented on Sep 26, 2019

Trott commented on Sep 26, 2019

anandsuresh commented on Sep 29, 2019

anandsuresh commented on Sep 30, 2019

anandsuresh commented on Sep 30, 2019

rvagg commented on Sep 30, 2019

jdwx commented on Oct 31, 2019

rvagg commented on Nov 1, 2019

mhdawson commented on Nov 1, 2019

jdwx commented on Nov 1, 2019

byalexandrepedrosa commented on Dec 10, 2019

github-actions commented on Oct 6, 2020

emaste commented on Oct 6, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions