This repository was archived by the owner on Nov 15, 2023. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 2.7k
R4R: prometheus exporter in substrate #4511
Closed
Closed
Changes from 1 commit
Commits
Show all changes
59 commits
Select commit
Hold shift + click to select a range
40762b6
Refactor rebase master prometheus_v0.3
nodebreaker0-0 472aca1
Milestone1: Final Version of v0.3
nodebreaker0-0 1974f94
no-std or warm compatibility issues, grapana-data -source code refere…
nodebreaker0-0 62a873d
Cargo.lock paritytech/master rebase
nodebreaker0-0 97e37e4
prometheus networking.rs del, grafana-data-source networking.rs pub e…
nodebreaker0-0 f3e444e
chore: reflect various feedback
nodebreaker0-0 8145df7
Spaces to tabs.
gavofyork fca71be
Replace grafana and tidy
expenses 855ceed
Add generics
expenses 70c6bcc
Add photo back
expenses 5941041
Re-fix spaces in primitives/consensus/babe/src/inherents.rs
expenses 7d9d341
Refactor rebase master prometheus_v0.3
nodebreaker0-0 52316ae
Milestone1: Final Version of v0.3
nodebreaker0-0 82cd8cf
no-std or warm compatibility issues, grapana-data -source code refere…
nodebreaker0-0 b052829
prometheus networking.rs del, grafana-data-source networking.rs pub e…
nodebreaker0-0 4e831e9
chore: reflect various feedback
nodebreaker0-0 0b15bc4
Replace grafana and tidy
expenses 8025361
Add generics
expenses fd081c9
Add photo back
expenses f00bb9b
Re-fix spaces in primitives/consensus/babe/src/inherents.rs
expenses 398da97
chore: revert this file back to paritytech/master inherents.rs.
nodebreaker0-0 84c458f
Add newline at EOF
expenses f931588
Merge remote-tracking branch 'nodebreaker/prometheus_v0.3' into ashle…
expenses 5796c85
Merge remote-tracking branch 'parity/master' into ashley-prometheus
expenses 2a45de8
Merge remote-tracking branch 'parity/master' into ashley-prometheus
expenses 8bae73b
Merge remote-tracking branch 'parity/master' into ashley-prometheus
expenses 1caa0f1
Merge remote-tracking branch 'parity/master' into ashley-prometheus
expenses ffb4746
Tidy
expenses 8a6e3c5
Use local registry
expenses 53c95de
fix typo
hskang9 23cb72e
chore: Apply review feedback
nodebreaker0-0 3d1634b
Merge remote-tracking branch 'nodebreaker/prometheus_v0.3' into ashle…
expenses 55ac4f8
Merge remote-tracking branch 'parity/master' into ashley-prometheus
expenses 3cfe43c
endpoint -> exporter
expenses 0af2369
fix readme
expenses 44f8c0e
Merge pull request #3 from paritytech/ashley-prometheus
nodebreaker0-0 d2bba61
Remove lazy_static, use ServiceMetrics struct instead
expenses 5d3d9a7
Merge pull request #4 from paritytech/ashley-prometheus
nodebreaker0-0 32c04b4
Merge branch 'paritytech/master' into prometheus_v0.3
mxinden 9bde830
Switch to using GaugeVecs
expenses 97faad6
Merge pull request #5 from mxinden/prometheus_v0.3
nodebreaker0-0 f64a482
chore: without nightly , edit README
nodebreaker0-0 5d56d93
Merge branch 'prometheus_v0.3' into ashley-prometheus
nodebreaker0-0 50f2928
block_height -> block_height_number
expenses 8edb710
Merge branch 'ashley-prometheus' of github.com:paritytech/substrate i…
expenses 6a25ea8
Switch to a ready_transactions_number gauge
expenses 18cf2be
Merge pull request #7 from paritytech/ashley-prometheus
nodebreaker0-0 bb388f5
Update utils/prometheus/src/lib.rs
hskang9 77ba252
no-prometheus flag add
nodebreaker0-0 5e327c2
/metrics url Input check
nodebreaker0-0 ccb3179
remove prometheus in Tracing
nodebreaker0-0 152176f
remove prometheus in Tracing
nodebreaker0-0 8de57c2
Merge branch 'master' into prometheus_v0.3
nodebreaker0-0 e76f200
chore: master code rebase edit
nodebreaker0-0 55428b7
gitlab-check-web-wasm edit code
nodebreaker0-0 d1003e6
From:from and cargo.lock update
nodebreaker0-0 5c2e085
Merge commit 'db1ab7d18fbe7876cdea43bbf30f147ddd263f94' into promethe…
nodebreaker0-0 b4b4432
with_prometheus_registry add background_tasks
nodebreaker0-0 ee421a6
Merge commit '419e5fd0026cfd528cd3b327789bb0a3a8215703' into promethe…
nodebreaker0-0 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Tidy
- Loading branch information
commit ffb4746ddc76139d67d6659e968fc0b0e1a2d7a8
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file not shown.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mxinden do you know of a way that we could implement this API? It seems like prometheus gauges etc are intended to created as static refs etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My guess is that you can't implement it without creating a macro.
Or you can simply make a static reference.
https://github.com/tikv/rust-prometheus/blob/master/examples/example_edition_2018.rs
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't yet understand fully how prometheus works, but to properly support this I think we would need to be able to:
With all targets currently in the code enabled, we can be approaching 200 measurements per second just across the runtime. Is this feasible to do while preserving timestamp (which we would need to explicitly provide for this receiver)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm. I'll have a go at this later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mattrutherford 200 measurements are not an issue for Prometheus. Prometheus can easily handle couple of mega bytes of metric data per scrape. See kubernetes/kube-state-metrics#498 for some numbers.
This sounds more like we want a Histogram, right?
Would it be possible to tackle the effort within
client/tracingas a follow up pull request? I would like to keep this one small. What do you think @mattrutherford @expenses @nodebreaker0-0?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree it's good to be clear on terminology, but that wasn't my main concern - only that if we're ripping out the Grafana server as part of the introduction of this Prometheus feature, is whether we'd lose this functionality to plot individual observations* - however that seems unavoidable based on what you said.
*For my use-case it's OK because I use the Telemetry Receiver in substrate-tracing to send the data to substrate-analytics which uses a PostgreSQL datasource, which in turn can be queried by Grafana (because I want to archive the data); however this is not trivial to set up, so the question is - do we really want to kill substrate Grafana server yet? It's a relatively new feature so maybe not many people use it, but I think it's something we should consider as part of our decision to remove it. Particularly as the current primary use-case for
substrate-tracingis for profiling and that might be handy for developers to have any easy way to look at this.Pinging @marcio-diaz @shawntabrizi @DarkEld3r in case you have opinion on this
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Imo the Grafana server in its current state isn't really good at doing anything. If Prometheus isn't going to work for tracing, we should probably find something that is. I'm happy enough to leave the Grafana server in for the moment until that happens.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, good to know - and just to be clear that I have no problem removing it, if it's not going to be a big detriment to people - just wanted to make sure everyone was aware of full implications
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think tracing other's nodes is not (should not) the purpose of the prometheus exporter. It is purely for monitoring operator's own nodes. Extracting information from nodes operated by others is not sounding right as a perspective of information privacy.
I guess tracing functionality should remain in telemetry, and it should be also "off as default" for privacy of node operators.
And storing and querying historical data for own nodes can be done on monitoring server(grafana), and it should not burden the node itself.
Please let me know if I misunderstood this context.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK - I can see what the misunderstanding is here and it warrants some more explanation - tracing is only capturing local data; which we then have the option to send somewhere, either Log (output tracing data via logger), Grafana server (built-in) or Telemetry. When using telemetry to send the tracing data, it is not expected (or desired) to send to the default telemetry url, so we override that via the cli, with eg:
--telemetry-url 'ws://localhost:8080 9'to send the data to an analytics server (often running on the same machine, but can be anywhere you want).