[SPARK-46812][PYTHON][TESTS][FOLLOWUP] Check should_test_connect and pyarrow to skip tests

[dongjoon-hyun]

What changes were proposed in this pull request?

This is a follow-up of SPARK-46812 to skip the tests more robustly and to recover PyPy CIs.

• https://github.com/apache/spark/actions/runs/8556900899/job/23447948557

Why are the changes needed?

• should_test_connect covers more edge cases than have_pandas.

• test_resources.py has Arrow usage too.
  

  spark/python/pyspark/resource/tests/test_resources.py

  Line 85 in 25fc67f

  df.mapInArrow(lambda x: x, df.schema, False, rp).collect()

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Manually tests with pandas and without pyarrow.

$ pip3 freeze | grep pyarrow
$ pip3 freeze | grep pandas
pandas==2.2.1
pandas-stubs==1.2.0.53

$ python/run-tests --modules=pyspark-resource --parallelism=1 --python-executables=python3.10
Running PySpark tests. Output is in /Users/dongjoon/APACHE/spark-merge/python/unit-tests.log
Will test against the following Python executables: ['python3.10']
Will test the following Python modules: ['pyspark-resource']
python3.10 python_implementation is CPython
python3.10 version is: Python 3.10.13
Starting test(python3.10): pyspark.resource.profile (temp output: /Users/dongjoon/APACHE/spark-merge/python/target/db9cb886-2698-49d9-a663-9b8bea79caba/python3.10__pyspark.resource.profile__8mg46xru.log)
Finished test(python3.10): pyspark.resource.profile (1s)
Starting test(python3.10): pyspark.resource.tests.test_connect_resources (temp output: /Users/dongjoon/APACHE/spark-merge/python/target/53f979bd-1073-41e6-99ba-8e787edc415b/python3.10__pyspark.resource.tests.test_connect_resources__hrgrs5sk.log)
Finished test(python3.10): pyspark.resource.tests.test_connect_resources (0s) ... 1 tests were skipped
Starting test(python3.10): pyspark.resource.tests.test_resources (temp output: /Users/dongjoon/APACHE/spark-merge/python/target/2b06c671-0199-4827-a0e5-f852a28313fd/python3.10__pyspark.resource.tests.test_resources__jis6mk9a.log)
Finished test(python3.10): pyspark.resource.tests.test_resources (2s) ... 1 tests were skipped
Tests passed in 4 seconds

Skipped tests in pyspark.resource.tests.test_connect_resources with python3.10:
      test_profile_before_sc_for_connect (pyspark.resource.tests.test_connect_resources.ResourceProfileTests) ... skip (0.002s)

Skipped tests in pyspark.resource.tests.test_resources with python3.10:
      test_profile_before_sc_for_sql (pyspark.resource.tests.test_resources.ResourceProfileTests) ... skip (0.001s)

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

should_test_connect covers more edge cases than have_pandas.

[dongjoon-hyun]

This test has df.mapInArrow(lambda x: x, df.schema, False, rp).collect() too.

[xinrong-meng]

LGTM thank you!

[dongjoon-hyun]

Thank you, @xinrong-meng !

[dongjoon-hyun]

Merged to master.

[dongjoon-hyun]

For the record, it's green now.

• https://github.com/apache/spark/actions/runs/8571815038