[DO NOT MERGE] Test NEW GA FAILURE

[yaooqinn]

In #32144

I encountered with https://github.com/yaooqinn/spark/runs/2344156574?check_suite_focus=true

2s
Run apache_spark_ref=`git rev-parse HEAD`
  apache_spark_ref=`git rev-parse HEAD`
  git fetch https://github.com/$GITHUB_REPOSITORY.git ${GITHUB_REF##*/}
  git merge --progress --ff-only FETCH_HEAD
  echo "::set-output name=APACHE_SPARK_REF::$apache_spark_ref"
  shell: /usr/bin/bash -e {0}
  env:
    MODULES_TO_TEST: core, unsafe, kvstore, avro, network-common, network-shuffle, repl, launcher, examples, sketch, graphx
    EXCLUDED_TAGS: 
    INCLUDED_TAGS: 
    HADOOP_PROFILE: hadoop3.2
    HIVE_PROFILE: hive2.3
    CONDA_PREFIX: /usr/share/miniconda
    GITHUB_PREV_SHA: a3a6c5e8401b6af21c329d71928a353432142349
    SPARK_LOCAL_IP: localhost
From https://github.com/yaooqinn/spark
 * branch                  SPARK-35044 -> FETCH_HEAD
fatal: Not possible to fast-forward, aborting.
Error: Process completed with exit code 128.

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

[yaooqinn]

https://github.com/yaooqinn/spark/actions/runs/748728586 still fails with the same error

Merge branch 'master' into SPARK-35044 or empty commit cause this? cc @HyukjinKwon I need help :)

[yaooqinn]

Run apache_spark_ref=`git rev-parse HEAD`
  apache_spark_ref=`git rev-parse HEAD`
  git fetch https://github.com/$GITHUB_REPOSITORY.git ${GITHUB_REF##*/}
  git merge --progress --ff-only FETCH_HEAD
  echo "::set-output name=APACHE_SPARK_REF::$apache_spark_ref"
  shell: /usr/bin/bash -e {0}
  env:
    MODULES_TO_TEST: core, unsafe, kvstore, avro, network-common, network-shuffle, repl, launcher, examples, sketch, graphx
    EXCLUDED_TAGS: 
    INCLUDED_TAGS: 
    HADOOP_PROFILE: hadoop3.2
    HIVE_PROFILE: hive2.3
    CONDA_PREFIX: /usr/share/miniconda
    GITHUB_PREV_SHA: 0000000000000000000000000000000000000000
    SPARK_LOCAL_IP: localhost
From https://github.com/gengliangwang/spark
 * branch                  tryNewGA   -> FETCH_HEAD
Updating 2974b70d1e..9536df22cc
Fast-forward
 sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

GITHUB_PREV_SHA: 0000000000000000000000000000000000000000 this looks weird but from a success build in @gengliangwang 's actions
GITHUB_PREV_SHA: a3a6c5e, this look correct but can not pass, I am not sure this is related

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41934/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41934/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41936/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41936/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41941/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41941/

[Yikun]

I met same problem on the 1st error fatal: Not possible to fast-forward, aborting., after some investagations, I found it is caused by yaooqinn:SPARK-35044 is not synced with apache/spark master, when we use git fetch and git merge to sync code, will raise the fast forward error[1].

Looks like we need a git rebase before fetch and merge?

[1] https://stackoverflow.com/questions/13106179/fatal-not-possible-to-fast-forward-aborting/43460847

[SparkQA]

Test build #137363 has finished for PR 32168 at commit 2e6cc14.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #137354 has finished for PR 32168 at commit b6d26b7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #137357 has finished for PR 32168 at commit f531cff.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Maybe we should remove fast forward option

[HyukjinKwon]

@Yikun can you open a PR that removes the -ff option in the sync? A bit of manual tests in your local is enough given that it breaks many PRs.

[HyukjinKwon]

github conflict detection is less strict to git rebase given my experience. If we can manage it without -ff option, that would be great. If that's difficult, we can do rebase for the time being.

[Yikun]

can you open a PR that removes the -ff option in the sync?

@HyukjinKwon Sure, will have a try....

(missing some word before, it's very wired)

[HyukjinKwon]

Thanks @Yikun! I sent a couple of emails related to this in private :-).