Ghost Batch Job Instances

[dcguim]

When creating SCDF task`s jobs with different names, even after cleaning up tasks through the dataflow interface, there are still references to undesired jobs in the BATCH_JOB_INSTANCE table. In this case, a batch job that is not displayed in the dataflow interface, will conflict with the job one is trying to execute and the java runtime will throw:

java.lang.IllegalStateException: Failed to execute CommandLineRunner

Nevertheless, upon deleting from BATCH_JOB_INSTANCE the conflicting job names the task executes successfully. For more information please refer to Corvin Newall's answer in SO:
https://stackoverflow.com/questions/50190952/failed-to-execute-commandlinerunner-spring-batch

[cppwfs]

Hello dcguim, can you provide :

• SCDF Version
• Database and version
• Stack trace of the deletion that left the orphans in the BATCH_JOB_Instance table.
• Also is this scenario repeatable? If so, can you share the steps to reproduce?

[dcguim]

Hello Glenn,

• DATAFLOW_VERSION=2.5.1.RELEASE
• image: mysql:5.7.25
  To reproduce please create a task and let it be stopped by SCDF, i.e. the jar which generated the batch task wasn't executed thoroughly. Clean up the problematic task, and the task`s job should still exist in the BATCH_JOB_INSTANCE table. If one tries to launch a new task with the same jobname as the orphan, the mentioned exception should be thrown.

[cppwfs]

When you say clean up the problematic task. Do you mean destroying the task definition or the task execution?

[dcguim]

I mean cleaning up the task execution (as in the GUI the "Actions" > "Clean up Task Executions(s)"), which should be already stopped by SCDF.

[cppwfs]

A couple more questions:
Is this exception happening everytime you do a cleanup?
Also does dataflow produce a stacktrace upon the deletion that fails (causes the problem)?
Can you provide the log entries around that deletion?
Is the exception and stack trace that you are experiencing in the task app match the one in the SO that you specify?

[dcguim]

Not every time, only when the JAR contains errors and is not successfully executed. It doesn't match completely. I will post the stack trace when I experience it again.

[dcguim]

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2020-09-17 11:41:59 ERROR Application run failed
java.lang.IllegalStateException: Failed to execute CommandLineRunner
	at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:787)
	at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:768)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:322)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215)
	at importer.batch.ServerBatchApplication.main(ServerBatchApplication.java:10)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:52)
Caused by: java.lang.NullPointerException
	at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.getExecutionContext(JdbcExecutionContextDao.java:111)
	at org.springframework.batch.core.repository.support.SimpleJobRepository.createJobExecution(SimpleJobRepository.java:136)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:366)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:99)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.batch.core.repository.support.AbstractJobRepositoryFactoryBean$1.invoke(AbstractJobRepositoryFactoryBean.java:181)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
	at com.sun.proxy.$Proxy68.createJobExecution(Unknown Source)
	at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:137)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
	at com.sun.proxy.$Proxy71.run(Unknown Source)
	at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.execute(JobLauncherCommandLineRunner.java:192)
	at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.executeLocalJobs(JobLauncherCommandLineRunner.java:166)
	at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.launchJobFromProperties(JobLauncherCommandLineRunner.java:153)
	at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.run(JobLauncherCommandLineRunner.java:148)
	at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:784)
	... 13 more
2020-09-17 11:41:59 INFO  Closing JPA EntityManagerFactory for persistence unit 'default'
2020-09-17 11:41:59 INFO  HikariPool-1 - Shutdown initiated...
2020-09-17 11:41:59 INFO  HikariPool-1 - Shutdown completed.

In the SCDF GUI:

In the mysql dataflow it shows:

Assuming I am trying to execute a second LoadingJob3 the only way to avoid this IllegalStateException is to delete job instance id 165 entry from the BATCH_JOB_INSTANCE table

[cppwfs]

This looks to be a Batch related issue based on the note above. Closing this issue for now. If in error reopen.