Add more visibility for dropped events

[Evedel]

This PR is bundled with a number of changes all regarding how event age is handled. Please let me know if you want to slice them differently or what your thoughts are in general.

1. Dropping event handling OnUpdate (diff link)

   • OnUpdate is called when there was a modification to an object. Events are only spawned/populated once and not modified later. Even if they were to be modified manually, their EventTime and LastTimestamp are going to stay the same as at an event creation (unless modified manually too, which goes against event logic), and so they are going to be discarded by kubernetes-event-exporter anyway
     • see https://github.com/resmoio/kubernetes-event-exporter/blob/master/pkg/kube/watcher.go#L74-L79

2. Adding warning and prometheus metrics for when an event is discarded (diff link)

   • Currently on main, if the eventAge > throttlePeriod => it is going to be silently dropped.
     Not only is it making debugging and choosing correct config options particularly hard (blind guessing), but also it is impossible to set promethues alarms for if a cluster has outgrown previously correct configs.
     Test case:
     • run two deployment of kubernetes-event-exporter
       • one is the exporter (kubeQPS:100, kubeBurst:500, throttledPeriod:5,logLevel: Info)
     • watch for only new events with k get events -A --watch-only > only-new-events.log
     • scale second deployments k scale deployment event-exporter-test --replicas 200
     • no signs of client-side throttling
     • no errors or warnings in logs
     • but the exporter only processed a small portion of events
       
   • With this PR and the same settings:
     • logs are clearly showing a problem and how to solve it
       
     • increase( events_discarded [1m]) can be used for alarms
       

3. Renaming config.throttlePeriod to config.maxEventAgeSeconds (diff link)

   • this is extremely confusing, as this parameter has nothing to do with the request throttling as shown in tests in (2)
   • config validation is added to accept both parameter names (for backward compatibility)
   • all examples/code/docs updated to use the new name

[Evedel]

Hello @mustafaakin, any chance you will be able to have a look?

[mustafaakin]

Sorry for late response. Your contribution is welcome, and thank you, we should handle it better. Can you fix the recent conflicts and send it again?

[Evedel]

Not a problem @mustafaakin. Ready for review now.