Skip to content

[SPARK-46302][TESTS] Skip maven daily testing as ivy uses some corrupted cache jar files #44208

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
panbingkun wants to merge 44 commits into from
Closed

[SPARK-46302][TESTS] Skip maven daily testing as ivy uses some corrupted cache jar files #44208

Show file tree
Hide file tree
Changes from 38 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
f0f7035
try fix maven test
panbingkun Dec 6, 2023
0b75a0f
try
panbingkun Dec 6, 2023
08e8859
try
panbingkun Dec 6, 2023
5fca2b7
try
panbingkun Dec 6, 2023
db9b07e
try
panbingkun Dec 6, 2023
742e29b
try
panbingkun Dec 6, 2023
953bc7e
try
panbingkun Dec 7, 2023
e3c6e8c
try
panbingkun Dec 7, 2023
4377bfc
try
panbingkun Dec 7, 2023
c128af6
try
panbingkun Dec 7, 2023
a7e1b12
try
panbingkun Dec 7, 2023
a0b9450
try
panbingkun Dec 7, 2023
77e94e9
fix maven tests
panbingkun Dec 7, 2023
224084e
Merge branch 'master' into try_fix_maven_test
panbingkun Dec 7, 2023
436c244
fix maven tests
panbingkun Dec 7, 2023
a9e146a
fix maven tests
panbingkun Dec 7, 2023
b7ae5cb
fix maven tests
panbingkun Dec 7, 2023
a57c512
try
panbingkun Dec 7, 2023
2fe75bd
try
panbingkun Dec 7, 2023
5a971b4
try
panbingkun Dec 7, 2023
dbcbd64
try
panbingkun Dec 7, 2023
b2fd3ee
try
panbingkun Dec 7, 2023
5acce38
Merge branch 'master' into try_fix_maven_test
panbingkun Dec 8, 2023
5221795
try
panbingkun Dec 8, 2023
dcbf922
try
panbingkun Dec 8, 2023
e4774cd
try
panbingkun Dec 8, 2023
c7561d2
try
panbingkun Dec 8, 2023
6f9a81c
try
panbingkun Dec 8, 2023
311a5dd
try
panbingkun Dec 8, 2023
66c78e9
try
panbingkun Dec 8, 2023
1ae55fd
try
panbingkun Dec 8, 2023
053c9a7
try
panbingkun Dec 11, 2023
54b7bea
Merge branch 'master' into try_fix_maven_test
panbingkun Dec 11, 2023
c79916e
[SPARK-46302][BUILD] Fix maven daily testing
panbingkun Dec 14, 2023
38094dd
Merge branch 'master' into try_fix_maven_test
panbingkun Dec 14, 2023
2035b91
[SPARK-46302][BUILD] Fix maven daily testing
panbingkun Dec 15, 2023
feb3932
[SPARK-46302][BUILD] Fix maven daily testing
panbingkun Dec 19, 2023
705d7e8
Merge branch 'master' into try_fix_maven_test
panbingkun Dec 19, 2023
ab06613
[SPARK-46302][BUILD] Fix maven daily testing
panbingkun Dec 19, 2023
aa37791
[SPARK-46302][BUILD] Fix maven daily testing
panbingkun Dec 19, 2023
9eb03f8
[SPARK-46302][TESTS] Skip maven daily testing as ivy uses some corrup…
panbingkun Dec 19, 2023
a70bf61
[SPARK-46302][TESTS] Skip maven daily testing as ivy uses some corrup…
panbingkun Dec 20, 2023
45aaecb
Merge branch 'master' into try_fix_maven_test
panbingkun Dec 20, 2023
941ab8e
[SPARK-46302][TESTS] Skip maven daily testing as ivy uses some corrup…
panbingkun Dec 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/workflows/build_and_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1098,3 +1098,9 @@ jobs:
cd ui-test
npm install --save-dev
node --experimental-vm-modules node_modules/.bin/jest

Copy link
Contributor Author

@panbingkun panbingkun Dec 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only for GA test.

maven-test:
Copy link
Contributor

@LuciferYang LuciferYang Dec 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fine to me, let's revert this change after test pass

Copy link
Contributor Author

@panbingkun panbingkun Dec 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay.

Copy link
Contributor Author

@panbingkun panbingkun Dec 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will revert this change, because the maven test has passed and the verification is successful.

Copy link
Contributor Author

@panbingkun panbingkun Dec 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

name: "Run Maven Test"
permissions:
packages: write
uses: ./.github/workflows/maven_test.yml
3 changes: 2 additions & 1 deletion .github/workflows/maven_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ on:
description: Additional environment variables to set when running the tests. Should be in JSON format.
required: false
type: string
default: '{}'
default: '{"SKIP_SPARK_RELEASE_VERSIONS": "3.3.3,3.4.2,3.5.0,master"}'
Copy link
Contributor

@LuciferYang LuciferYang Dec 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need for master

Copy link
Contributor

@LuciferYang LuciferYang Dec 19, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and default should be '{}' or '', we should pass it from build_maven.yml

jobs:
# Build: build Spark and run the tests for specified modules using maven
build:
Expand Down Expand Up @@ -115,6 +115,7 @@ jobs:
HIVE_PROFILE: ${{ matrix.hive }}
SPARK_LOCAL_IP: localhost
GITHUB_PREV_SHA: ${{ github.event.before }}
SKIP_SPARK_RELEASE_VERSIONS: ${{ fromJSON(inputs.envs).SKIP_SPARK_RELEASE_VERSIONS }}
Copy link
Contributor

@LuciferYang LuciferYang Dec 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This option should not be needed, the Run tests section has already been deserialized for env

spark/.github/workflows/maven_test.yml

Lines 172 to 173 in 0b8e099

- name: Run tests
env: ${{ fromJSON(inputs.envs) }}

steps:
- name: Checkout Spark repository
uses: actions/checkout@v4
Expand Down
5 changes: 4 additions & 1 deletion sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -249,6 +249,8 @@ class HiveExternalCatalogVersionsSuite extends SparkSubmitTestUtils {
}

object PROCESS_TABLES extends QueryTest with SQLTestUtils {
private val skipReleaseVersions =
sys.env.getOrElse("SKIP_SPARK_RELEASE_VERSIONS", "").split(",").toSet
val isPythonVersionAvailable = TestUtils.isPythonVersionAvailable
val releaseMirror = sys.env.getOrElse("SPARK_RELEASE_MIRROR",
"https://dist.apache.org/repos/dist/release")
Expand All @@ -263,7 +265,8 @@ object PROCESS_TABLES extends QueryTest with SQLTestUtils {
.filter(_.contains("""<a href="spark-"""))
.filterNot(_.contains("preview"))
.map("""<a href="spark-(\d.\d.\d)/">""".r.findFirstMatchIn(_).get.group(1))
.filter(_ < org.apache.spark.SPARK_VERSION).toImmutableArraySeq
.filter(_ < org.apache.spark.SPARK_VERSION)
.filterNot(skipReleaseVersions.contains).toImmutableArraySeq
} catch {
// Do not throw exception during object initialization.
case NonFatal(_) => Nil
Expand Down