Maintenance tasks for repository lmms

[jasp00]

This adds an hourly cron job to maintain the repository lmms. Is an access to GitHub set up already? A commit preparing the affected files should be done first.

[tresf]

Hourly may be overkill, don't you think?

[jasp00]

Hourly may be overkill, don't you think?

It is only a git pull and the scripts are quick. I could avoid the scripts though, if there were no errors.

[jasp00]

I could avoid the scripts though, if there were no errors.

Done.

[tresf]

• Current cron file naming convention looks like this: update_community_feeds. Please replace dashes with underscores.
• The file name lmms-tasks should be more descriptive, such as git_maintenance_tasks
• Although the intent of maintenance-tasks-done seems to be to make sure it runs at least once, the code suggests it's not needed; it'll hit else on the if [ -e lmms ] block.
• To be safe, git clone [...] should explicitly specify a branch. I recommend master.
• Already up-to-date. should set terminal to English to make sure this parses the English output (e.g. LC_ALL=C)

I like the thought that went into the cron/tasks structure and I don't mean to be overly critical, but I feel this could all be in one script.

For example... here's the entire task in 34 lines with comments. I haven't tested the below code but in addition to the above points... When run interactively, it should be apparent what it's doing for support purposes.

#!/bin/sh

# Exit on any non-zero return codes; force en_US locale
set -e && LC_ALL=C
mkdir -p ~/lmms-repo && cd ~/lmms-repo

if [ -e lmms ]; then
	cd lmms && PULLRES=$(git pull)
	if [ "$PULLRES" = "Already up-to-date." ]; then
		exit
	fi
else
	git clone -b master https://github.com/LMMS/lmms && cd lmms
fi

# Update CONTRIBUTORS
git shortlog -sne | cut -c 8- > doc/CONTRIBUTORS

# Update PROJECT_YEAR 
YEAR=$(date -u +%Y)
while true; do
	if [ "$(git log -n 1 --since=$YEAR-01-01T00:00:00Z)" ]; then
		break
	fi
	YEAR=$((YEAR - 1))
done

sed -e "/^SET(PROJECT_YEAR / c \SET(PROJECT_YEAR $YEAR)" -i CMakeLists.txt

# Commit changes, if any
if [ "$(git diff)" ]; then
	git add -u && git commit -m "Maintenance tasks"
	git push origin master
fi

[tresf]

Is an access to GitHub set up already?

No, not yet. lmmsservice is our GitHub service account. I'd be happy to set it up with a pointer in the right direction. Alternately, we can add your public key to the server if you prefer to test it all out yourself first.

[jasp00]

• Although the intent of maintenance-tasks-done seems to be to make sure it runs at least once, the code suggests it's not needed

It is needed to run the tasks at most once, otherwise the tasks would run each hour.

I feel this could all be in one script.

The point is to easily add, enable and debug tasks; they are like standalone plug-ins. Each task should have its own commit actually. More tasks are expected to be developed.

we can add your public key to the server

This is an option, but not necessary.

I'd be happy to set it up with a pointer in the right direction.

Option 1, with SSH:

1. Generate 4096-bit RSA SSH keys.
2. Upload the public key to GitHub.
3. Configure ~/.ssh/config.

Option 2, with password:

1. Configure ~/.netrc.

[tresf]

It is needed to run the tasks at most once, otherwise the tasks would run each hour.

If it only commits changes, I don't see how this statement is valid.

The point is to easily add, enable and debug tasks; they are like standalone plug-ins. Each task should have its own commit actually.

That's an opinion. As the site maintainer, I won't merge it in this condition. If another maintainer disagrees you can work with them.

More tasks are expected to be developed.

This is speculative.

Option 1, with SSH:

Thanks. I'll post back to this thread once setup.

[jasp00]

The goal is scalability.

otherwise the tasks would run each hour.

If it only commits changes, I don't see how this statement is valid.

The tasks would not make any commit, but they would still run (do you remember?).

The point is to easily add, enable and debug tasks;

That's an opinion.

Why would you not want a system that is based on run-parts, which is easy to debug single tasks, to disable with a file rename, to expand with independent executables, etc.? Why would you want a system that is not scalable?

More tasks are expected to be developed.

This is speculative.

I mentioned some possible tasks: remove executable bits, optimize PNG images, format code, etc. Do you want me to add another task?

[tresf]

The tasks would not make any commit, but they would still run (do you remember?).

What you're referencing is the question if hourly was overkill. Your argument I felt was valid.

jasp00 wrote: "It is only a git pull and the scripts are quick. I could avoid the scripts though, if there were no errors."

Still, I don't like magic lock files. They're bad design as they require manual cleanup when things fail. Last, I don't even understand the purpose of this one. I can't see a logical scenario when it's warranted.

I mentioned some possible tasks: remove executable bits, optimize PNG images, format code, etc. Do you want me to add another task?

Very good points. Sorry for the delay on the SSH push config. Will set that up ASAP.

[jasp00]

Still, I don't like magic lock files. They're bad design as they require manual cleanup when things fail.

There is no cleanup, lock files do not need to be removed. These locks are released when the process finishes. AFAIK, this is the best way to have only one instance per service.

Last, I don't even understand the purpose of this one. I can't see a logical scenario when it's warranted.

It is defensive coding. The case is when the process lasts more than an hour somehow. This may happen if networking goes slow, if the server is heavy loaded, if a task is deadlocked, if we include 60 one-minute tasks, etc. If this happens, we do not want a second instance to interfere with the first one.

I should add a protection against merge conflicts.

[jasp00]

I should add a protection against merge conflicts.

In the future, it is very unlikely right now.

[tresf]

I should add a protection against merge conflicts.

In the future, it is very unlikely right now.

As is a 1-hour-long merge.

[jasp00]

As is a 1-hour-long merge.

1-hour-long process actually; I know it is unlikely. I would implement a protection against merge conflicts, but I have more pressing issues right now. The point is I would not reject an improvement.

[tresf]

@jasp00 SSH authentication is complete.

• Pushes will authenticate automatically; If for some reason they don't, we can prepend ssh-agent to the git push command.
• Pushes must go to ssh instead of the default HTTP push.

  # example
  git remote set-url origin [email protected]:lmms/lmms.git

• Here is a preview of what the push will look like: tresf/lmms@c7f025f

[jasp00]

We should test with a branch first, then switch to master, do you agree?

[tresf]

We should test with a branch first, then switch to master, do you agree?

Sure.

[jasp00]

Ready.

[tresf]

Should kick off in 28 minutes...

@jasp00 why are the scripts in /dev and /tasks and not /cron ?

[tresf]

Answering my own question, dev was already there. I'm still confused as to the need for the new root directory /tasks. Seem like /dev could have been the home for tasks.

[tresf]

Ok... @lukas-w put the other script in /dev for a different purpose -- to help him test locally. We need to move stuff around. /cron can house everything and /dev should go back to just developer helpers/tools.

[jasp00]

I'm still confused as to the need for the new root directory /tasks.

I chose /tasks to be independent of the trigger, either cron job or webhook.

We need to move stuff around.

Sure.

There are no commits in the test branch. Are you sure cron is running? Should I prepend ssh-agent?

[tresf]

Cron is running, there was just an authentication problem which has been fixed d2d6704.

Note, the task was run interactively for testing purposes; the commits (LMMS/lmms@b990b18, LMMS/lmms@bf0e650), were done with the interactive shell and on initial clone (haven't tested pull) but initial tests pass.

[tresf]

@jasp00 should we consider appending git checkout $BRANCH to force us back onto master once ready?

[jasp00]

Sure, #214.

[tresf]

I can't find your most recent comments about the job failing but interactively tests pass, then when I scheduled the hourly to run now, it reverted the branch to the past somehow. https://github.com/LMMS/lmms/blob/maintenance-test/doc/CONTRIBUTORS

[tresf]

Ok... @jasp00 I think we've hit a nuance with git add.

http://stackoverflow.com/questions/348170/how-to-undo-git-add-before-commit

I can reproduce the flapping behavior of HEAD and reset the file. We can either try moving git add inside the conditional block, or try git reset <file> to remove it although there's a very highly up-voted answer that instead says we should use git rm --cached <file>.

[jasp00]

Ok... @jasp00 I think we've hit a nuance with git add.

I do not think so, git add works as intended. See #215 (comment).