Edge case: ambiguous failure on HPC with SGE

[Tassadaar]

Hi, thank you for rewriting BRAKER3 and with such speed!

I have a rather niche case that may be of relevance.

What happened?

I ran with --executor slurm flag on a HPC with SGE.

run_braker4.sh

snakemake \
    -s /scratch3/jasons/protist_gene_prediction/software/BRAKER4/Snakefile \
    --executor slurm \
    --default-resources slurm_partition=batch mem_mb=120000 \
    --cores 48 \
    --jobs 48 \
    --use-singularity \
    --singularity-prefix .singularity_cache \
    --singularity-args "-B /scratch3"

run_braker4.sh.e5475

...
total                             41
Select jobs to execute...

Execute 3 jobs...

No SLURM account given, trying to guess.

No SLURM account given, trying to guess.

No SLURM account given, trying to guess.

Subsequent run locked up without --executor flag.

run_braker4.sh

snakemake \
    --cores 40 \
    -s /scratch3/jasons/protist_gene_prediction/software/BRAKER4/Snakefile \
    --use-singularity \
    --singularity-prefix .singularity_cache \
    --singularity-args "-B /scratch3"

run_braker4.sh.e5482

Assuming unrestricted shared filesystem usage.
Building DAG of jobs...
LockException:
Error: Directory cannot be locked. Please make sure that no other Snakemake process is trying to create the same files in the following directory:
/scratch3/jasons/protist_gene_prediction/data/pipeline/blastocystis_st2/braker4/ab_initio
If you are sure that no other instances of snakemake are running on this directory, the remaining lock was likely caused by a kill signal or a power loss. It can be removed with the --unlock argument.

Subsequent runs failed ambiguously with --unlock flag.

run_braker4.sh

snakemake \
    --cores 40 \
    -s /scratch3/jasons/protist_gene_prediction/software/BRAKER4/Snakefile \
    --use-singularity \
    --singularity-prefix .singularity_cache \
    --singularity-args "-B /scratch3" \
    --unlock

run_braker4.sh.e5485

Building DAG of jobs...
Unlocked working directory.

What fixed it?

Completely deleting .snakemake/.

Closing thoughts:

I think this has to do with BRAKER4's ability to resume the previous failed run, or it could be a snakemake issue. Granted, I should have known which job scheduler my HPC is using, but I thought it would be helpful to document this rather unintelligent issue in case it comes up for another poor bloke in the future.

Follow-up question:

I noticed that there is a SGE-compatible plugin for snakemake, will this actually work as seamlessly for BRAKER4?

P.S., I don't think it's that relevant, but the SGE version is SGE r247.8.1.9-236-g8b28e5d.

[KatharinaHoff]

I cannot test SGE on my end because I currently do not have access to an SGE HPC, anymore. If you report it works, we can document that.

[Tassadaar]

I'm happy to report that the SGE-compatible plugin for snakemake does work!

mkdir -p sge_logs

snakemake \
    --executor cluster-generic \
    --cluster-generic-submit-cmd "qsub -cwd -V -o sge_logs -e sge_logs" \
    --default-resources \
    --cores 40 \
    --jobs 40 \
    --snakefile /scratch3/jasons/protist_gene_prediction/software/BRAKER4/Snakefile \
    --use-singularity \
    --singularity-prefix .singularity_cache \
    --singularity-args "-B /scratch3" \

[KatharinaHoff]

Thank you! I documented this.