Refactor node-mixin as node-observ-lib

[v-zhuravlev]

Similar to windows-observ-lib, node-observ-lib is created, using many panels from commonlib:. It also refactors everything from node-mixin by using grafonnet (v10 schema).

Node observability lib would allow:

• Easily read and maintain dashboards as code
• Easily modify any dashboard, panel, target, variable... before rendering final result
• Change instanceLabels, groupLabels from default job,instance to use dashboards in environments with extra or custom labels
• Instantiate node dashboards more than once in a single environment (Uid and filteringSelector are used to avoid conflicts).

To learn more about suggested mixin packaging format (observ-lib) you can check dataless example here.

What is recreated/moved from old node-mixin:

• Dashboards except USE.
• All alerts
• All recording rules

Added/Updated:

• Dashboards:
  • Fleet overview
  • Overview dashboard (added inventory row)
  • Drill down dashboards:
    • Memory
    • CPU and system
    • Network (Interfaces & Sockstat / Netstat)
    • Disks and filesystem
    • Logs (only added if enableLokiLogs:true)
  • MacOS overview (added inventory row)
  • MacOS logs (only added if enableLokiLogs:true)
• Prometheus annotations (for all dashboards):
  • Reboot
  • OOM kill detected
  • Kernel update
• New loki annotations (only added if enableLokiLogs:true)
  • Service failed
  • Critical system event
  • Session opened
  • Session closed
• Alerts:
  • filesystem alerts are moved into separate alerts group to avoid hitting limits for number of alerts allowed in group (in Grafana Mimir)
• Title case is used across all recreated panels and dashboard names

[gaetanars]

Just a little question : why don't you render dashboards with .json suffix?

[v-zhuravlev]

Just a little question : why don't you render dashboards with .json suffix?

Nothing specific, It was a little bit more convenient to reference dashboards by key (dashboards.overview, dashboards.network etc..)..

[SuperQ]

If this is intended to replace the node-mixin, should we remove the old mixin?

[v-zhuravlev]

Just a little question : why don't you render dashboards with .json suffix?
@gaetanars, I think you are right, it is expected for end files to have .json suffix. Brought it back.

If this is intended to replace the node-mixin, should we remove the old mixin?
@SuperQ , we could, but we need to either to reimplement use dashboards as well, or agree that they could be dropped.

[v-zhuravlev]

Just let me know, if what do you think I should do to merge this :)

[v-zhuravlev]

@SuperQ ,Hi, sorry for bumping, is there anything blocking to move this forward? thanks

[discordianfish]

Can it produce the same dashboards as the old mixin? Then we could remove the old one. If not, I dont know. We certainly dont want to support both..

[v-zhuravlev]

The only difference is USE dashboards, there are not present in new set:

USE dashboards (node-rsrc-use.json, node-cluster-rsrc-use.json,node-multicluster-rsrc-use.json)

[discordianfish]

@v-zhuravlev Could you add the USE dashboards as well? Then I'm open to replace the old mixins with this.

[v-zhuravlev]

@v-zhuravlev Could you add the USE dashboards as well? Then I'm open to replace the old mixins with this.

Sure, will add them shortly

[v-zhuravlev]

Use dashboard migration:
Will add commits shortly.

[v-zhuravlev]

@v-zhuravlev Could you add the USE dashboards as well? Then I'm open to replace the old mixins with this.

1. Added use dashboards and merged node-mixin and node-observ-lib folders.
2. Synced the latest PRs (AIX etc)
3. Included Add NodeSystemdServiceCrashlooping alert to mixin #3039
   @discordianfish
   Could you take a look please?

USE:

USE cluster:

Use mutlicluster:
Before:

After:

[v-zhuravlev]

@discordianfish, hi! Any chance to check it? thanks!