Add JobParametersIncrementer implementation based on a DataFieldMaxValueIncrementer

[spring-projects-issues]

Gregory D. Hopkins opened BATCH-2074 and commented

Due to problems listed in the linked StackOverflow Q, I created an implementation of org.springframework.batch.core.JobParametersIncrementer based on Spring JDBC's org.springframework.jdbc.support.incrementer.DataFieldMaxValueIncrementer.

It looks like org.springframework.jdbc.support.incrementer.DataFieldMaxValueIncrementer is used several places in the Spring Batch project, but I could not find any that provided linkage to org.springframework.batch.core.JobParametersIncrementer.

Hopefully, others will find this useful.

---

Reference URL: http://stackoverflow.com/questions/17975917/spring-batch-runidincrementer-not-generating-next-value

Attachments:

• DataFieldMaxValueIncrementerJobParametersIncrementer.java (1.21 kB)

1 votes, 4 watchers

[spring-projects-issues]

Michael Minella commented

Can you create this as a pull request on Github? Please be sure to add a unit test for your component and fill out the contributor's license agreement if you have not already.

If you have any questions about submitting a pull request, you can read more about it here: https://github.com/SpringSource/spring-integration/blob/master/CONTRIBUTING.md

[spring-projects-issues]

Gregory D. Hopkins commented

If at home, I would have gone the GitHub route... My enterprise group has GitHub blocked. :( Thanks!

[spring-projects-issues]

Gregory D. Hopkins commented

By the way, any comment on the StackOverflow Q?

[spring-projects-issues]

Michael Minella commented

In what you have in the StackOverflow question, I don't see anything obvious. You say that you are 100% sure that you are not accidentally kicking off the same job multiple times at once? If you have a test case that can reproduce the issue, I'd be more than happy to dig a bit deeper...

[spring-projects-issues]

Alexis SOUMAGNE commented

I've already implemented something similar here : https://github.com/almex/Raildelays/blob/master/batch/src/main/java/org/springframework/batch/core/JobParametersSequenceIncrementer.java

If you're interested I can create a pull request.

[spring-projects-issues]

Michael Minella commented

Moving this to 4.2 since we need a test written.

[spring-projects-issues]

Mahmoud Ben Hassine commented

@Gregory D. Hopkins Can you open a PR on github with the attached file? It would be great if you add a test too. We will review it on github and consider merging it for v4.2. Thank you upfront.

[fmbenhassine]

I polished the attached file and added a test (see a333c16).

For future reference, here is a summary of the cases where the new DataFieldMaxValueJobParametersIncrementer has an added value compared to the RunIdIncrementer:

1. Un-ordered database sequences

Some database providers do not guarantee the order of numbers generated by sequences (like Oracle where NOORDER is the default). This means job instance IDs generated by BATCH_JOB_SEQ might not be in order. However, Spring Batch assumes that the last instance is the one with the highest ID. In this case, the incrementer might return the run.id of a previously completed instance (mistakenly leading to JobInstanceAlreadyCompleteException). Here is an example:

Job instance	1	22	23	2	3
run.id	1	2	3	4	4

In this example, when creating the job instance 2, Spring Batch will calculate the last instance (ie 23) and pass its run.id (which is 3) to the incremnter, which in turn returns 4. The same thing happens when creating job instance 3 (the max is always 23 here) and the incrementer will also return 4 leading to a JobInstanceAlreadyCompleteException at the time of launching the job. The new incrementer would return a different value thanks to the underlying DataFieldMaxValueIncrementer.

2. Concurrent access to the incrementer

The job parameters incrementer might be called concurrently when:

• Multiple threads call JobOperator#startNextInstance at the same time
• Multiple processes call CommandLineJobRunner with -next option at the same time

In these cases, those threads/processes might end-up with the same set of next parameters returned by the incrementer (since they might calculate the same last instance and pass its parameters to the incrementer). This leads to a JobExecutionAlreadyRunningException being thrown by the threads/processes that were not able to create the job instance (only one of them will succeed, others will fail). Since most DataFieldMaxValueIncrementer implementations are thread-safe, the new incrementer should guarantee the uniqueness of generated values (and even if the implementation is not thread-safe, this should be guaranteed at the database level).