Capture and report systemd unit failures by default to the MCS

[cgwalters]

We've debated this a few times - who watches host failures? I think we should do something like have a small systemd unit machine-config-daemon-host-monitor.service which watches systemd and if any unit fails, does a POST to the MCS.

Yes this implies a MCS write endpoint, not just read. We could start by just dumping the data into the pod logs.

This is also related to coreos/ignition#585 which we'd also want a MCS endpoint for.

[cgwalters]

Also related to this, something in the platform really must start watching for kernel oopses.

Also xref https://github.com/kubernetes/node-problem-detector

[cgwalters]

Now that #1766 landed - we can make this be part of the MCD itself quite easily.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[cgwalters]

/lifecycle frozen
We really need this. See e.g. https://bugzilla.redhat.com/show_bug.cgi?id=1906057 where the real problem is lost in the black hole somewhere in Ignition or firstboot.

I'd been hoping we could reuse openshift/enhancements#443 to authenticate the MCS endpoint for failure report submission.