Native add-on compile build & cache service

[rvagg]

It turns out that @orangemocha and I have been thinking about a very similar solution to the native add-on compile problem.

1. Accept queries for binaries of packages with the following parameters:
   • Package name (e.g. bignum, leveldown, serialport)
   • Package version (e.g. 1.0.2)
   • Platform (e.g. linux, win32, darwin)
   • Architecture (e.g. x86, x64, armv7)
   • Node ABI version (i.e. process.versions.modules, a.k.a NODE_MODULES_VERSION)
2. If no matching binary exists and the combination is acceptable build it
3. Serve the binary or serve a rejection for <reason>

Some additional thoughts about getting something like this working:

• Windows is arguably the most difficult platform with the most unknowns to get this working on. OSX is similar for various reasons but probably not as difficult as Windows and doesn't present as many challenges to users. So we should pick off Windows as a first target and ignore the rest until we have something that works for Windows, then we can rapidly apply the same model to other platforms.
• Containers, jails and zones are ideal where available, VMs with snapshots otherwise. It's important that we provide isolation between builds and isolation of the build system for security, stability and overall integrity of the system.
• Build resource requirements would be bursty, mainly coinciding with new releases of Node and new releases of the supported packages. Building elasticity into the system would be sensible.
• Azure would be a good target for seeking resources for a first-pass at this, if we can get them on board we could spin up a proof of concept fairly quickly I think. It's in Microsoft's interests so we just need to connect that with Azure.
• OSX resources are tricky, we can't get this as easily from an IaaS but we could either buy more Mac Minis and store them in a partner's network (Voxer does this atm, I do this personally for ARM). We could also investigate using Foundation resources to buy from one of the pseudo IaaS OSX providers.
• Linux and other unixes are fairly easy to add but are also the platforms with the least pain (apt-get install build-essential && bingo!). Piggy-backing @nodejs/build resources is possible but we're already close to saturated for usage so we'd have to spin up more to handle this and we may be pushing friendships with sponsors.
• Linux ARM is the trickiest of the unixes because it's not as simple to compile often because of resource constraints and the style of deployment. We could piggy-back the existing @nodejs/build ARM resources here and seek donations from the community (ideal) or funds from the Foundation (if necessary).
• We can't support all native add-ons, consider node-canvas as one case where there are system dependencies that are necessary to both compile and use it. Unless a native add-on can ship its dependencies via npm it's going to be difficult for us to support it.
• We should consider a two-way opt-in system whereby we are whitelisting packages that we are willing to support, and those packages are opting in with their install system in the way they use node-pre-gyp now.
• node-pre-gyp has an evolved model that seems to be working well, we should adopt as much of that as possible and possibly piggy-pack it to provide the implementation (i.e. node-pre-gyp could be the client for this system).

/cc @nodejs/build, @springmeyer, @othiym23, @nebrius (pls unsubscribe if you don't want to be involved in this discussion, I'm just guessing)

[nebrius]

I think this sounds really great, I hope we can get some forward momentum on it.

Are you envisioning the query described in step 1 as a part of the NPM install process, i.e.

1. user types npm install
2. the NPM client queries this service running on Azure
3. the service returns the binary if it has the binary cached
4. if not, the service builds it on the fly, if it has the correct build configuration and caches that build
5. otherwise it returns an error to the NPM client

Did I summarize that correctly?

[othiym23]

This sounds fantastic and aligns with a bunch of stuff on npm's road map. How can npm help?

[jbergstroem]

This also makes the assumption that node/io.js has been built a specific way (i.e. no shared libraries).

Edit: as well as linked dependencies being met (mysql, postgres, nan, etc)

[orangemocha]

Really looking forward to making this happen 👍 The native module compile issue has been a thorn on my side for quite some time.

I will start by looking at the Azure side of things. That is, unless someone knows of a cloud provider that can virtualize OSX/Darwin.

[orangemocha]

@othiym23 when a module is published to npm, npm makes no guarantees that it doesn't contain any malicious code, right? I suppose the same guarantees (or lack thereof) will have to apply to the prebuilt version.

[rmg]

+1 on using foundation money to pay for an OS X pseudo IaaS since it also solves the trust problem with access to private networks.

[rmg]

Three groups come to mind that would benefit from a service like this, but one of them might not even be able to use it.

1. Slow environments like ARM where any reduction in work during module install would be appreciated.
2. Users who don't have a build environment because it is too difficult to install/configure.
3. Users who don't have a build environment because they are prohibited from installing one.

The 3rd group, unfortunately, also seems the least likely to be allowed to use such a service. Is there anything that can be done to make these binaries trustworthy to those users? Some combination of signed npm packages and signed binaries maybe? @othiym23 I didn't see anything like that (signing) on the npm roadmap (I was only skimming), should it be?

[indexzero]

Many moons ago @bmeck and I wrote a few modules around this with this very use-case in mind. In particular it deals with cross-platform, multiple architecture, and multiple node versions from a single source. Might be useful for these current efforts.

[cjbj]

As a native add-on owner, these are issues/requirements I see:

• Improvements to/over npm for download of non-npm'able required
  components, e.g. platform specific third party database client
  libraries. Scripting this with https://docs.npmjs.com/misc/scripts
  gets messy.
• ability to query/match versions of dependent 3rd party libraries.
  I.e. extend the queries @rvagg listed in
  Native add-on compile build & cache service #151 (comment) to include
  at least one more field, for example which database client library
  version the add-on should be compiled with.
• The node-gyp requirement for Python 2.7 is going to be an ongoing
  issue for native extensions, unless you replace this part of the
  process too.
• Making extension licenses visible before install (or better, support
  click-through licenses now or in future) would make the big
  corporate ecosystem happy. Users and their bosses like clear
  expectations about liability.
• Potential integration with external compile/build services that host
  extensions you decide not to, or services that are hosted inside
  company firewalls (e.g. replicas to cut down external bandwidth
  usage, or with private modules). These would presumably all be
  running the same software.

[rvagg]

Thanks @cjbj, great input on all counts, particularly re integration with alt build services. Can you elaborate on your point about licenses though and how it related specifically to this effort?

[cjbj]

@rvagg Pretty much any software you will serve has a license. Lawyers like click-throughs for lawyerly reasons and will quote precedences ad infinitum.

Allowing add-ons to choose to have a click-through before their binary is downloaded will let owners satisfy any legal restrictions, either for the add-on code itself, or for its third-party dependencies.

If click-through support isn't feasible to implement, then the second best option is to make sure the add-on license(s) are visible or accessible prior to serving any software. Either the full licence text or SPDX identifier should be accessible.

Of all the points in my previous post, having click-through support is the most important to me.

[nebrius]

@cjbj just to clarify, is click-through a feature you would like to see in the NPM client?

[cjbj]

@nebrius absolutely yes.