Skip to content

🐛 Manager.Elected() should beclosed after runnables are started #1354

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jan 21, 2021
Merged

🐛 Manager.Elected() should beclosed after runnables are started #1354

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
🐛 Manager.Elected() should beclosed after runnables are started 
During debugging a weird issue with some tests failing if they were
running too fast. In other words, calling Start() and Close() on
a manager too fast throws an error where the informers haven't been able
to sync, which then makes Start() fail with an error.

In an effort to improve this behavior, tried to use Elected() to wait
for leader election to start, which also waits for the cache.

During some debugging, this issue happened again and upon digging a bit
more it seems that the channel was closed before starting the runnables
in some cases.

This change reorders the close on cm.elected, which should fix the above
issue.

Signed-off-by: Vince Prignano <[email protected]>
  • Loading branch information
@vincepri
vincepri committed Jan 21, 2021
commit c9e1c103ee524517fd1e1eacf7c753ea2310f6df
4 changes: 2 additions & 2 deletions pkg/manager/internal.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -473,8 +473,8 @@ func (cm *controllerManager) Start(ctx context.Context) (err error) {
}
} else {
// Treat not having leader election enabled the same as being elected.
cm.startLeaderElectionRunnables()
close(cm.elected)
Copy link
Member

@alvaroaleman alvaroaleman Jan 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vincepri this is consistent today. We first close the channel, the we start the Runnables.

Also, is it intended to make the cm.StartLeaderElectionRunnables synchronous?

Copy link
Member Author

@vincepri vincepri Jan 21, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that was an intentional change

Copy link
Member

@alvaroaleman alvaroaleman Jan 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah now I get it, this is abusing the fact that startLeaderElectionRunnables synchronously waits for the cache to be synced. A tad implicit and has a good chance to be accidentally broken in the future again, but seems reasonable

Copy link
Contributor

@estroz estroz Jan 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now leader and leaderless starting of pre-election runnables is the same, that makes sense to me.

go cm.startLeaderElectionRunnables()
}
}()

Expand Down Expand Up @@ -632,8 +632,8 @@ func (cm *controllerManager) startLeaderElection() (err error) {
RetryPeriod: cm.retryPeriod,
Callbacks: leaderelection.LeaderCallbacks{
OnStartedLeading: func(_ context.Context) {
close(cm.elected)
cm.startLeaderElectionRunnables()
close(cm.elected)
},
OnStoppedLeading: cm.onStoppedLeading,
},
Expand Down