GH-48448: [Python] Implement Alphanumeric and Surrogate text in the random schema generator #48449

HyukjinKwon · 2025-12-11T03:58:53Z

Rationale for this change

To test various cases of field names:

arrow/python/pyarrow/tests/strategies.py

Line 49 in 6456944

# TODO(kszucs): alphanum_text, surrogate_text

It was introduced from 9da4584

What changes are included in this PR?

This PR implements Alphanumeric and Surrogate text in the random schema generator

Are these changes tested?

Yes I tested them via:

PYARROW_TEST_HYPOTHESIS=1 pytest -xvs pyarrow/tests/test_strategies.py::test_fields --hypothesis-show-statistics

Are there any user-facing changes?

No, test-only.

GitHub Issue: [Python] Implement Alphanumeric and Surrogate text in Random schema generator #48448

HyukjinKwon · 2025-12-11T03:59:09Z

python/pyarrow/tests/strategies.py

        max_codepoint=0x7E
    )
 )



I referred to https://www.unicode.org/reports/tr44/#General_Category_Values and https://datatracker.ietf.org/doc/html/rfc2781#section-2.1

github-actions · 2025-12-11T03:59:25Z

⚠️ GitHub issue #48448 has been automatically assigned in GitHub to PR creator.

AlenkaF · 2025-12-12T09:22:04Z

@github-actions crossbow submit -g python

AlenkaF

LGTM though I do not have lots of knowledge to be critical about the selection for min and max code points. Will ask for another review (and extended CI) before merging, cc @raulcd @rok

HyukjinKwon · 2025-12-12T09:24:15Z

(just for a bit of context, I am fixing hypothesis at #48460 as it's broken, and @raulcd guided me to work on this at #48443 (review) 👍)

github-actions · 2025-12-12T09:24:56Z

Revision: de8e939

Submitted crossbow builds: ursacomputing/crossbow @ actions-15cc39ef2c

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-1.3.4-numpy-1.21.2
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.12-pandas-latest-numpy-1.26
test-conda-python-3.12-pandas-latest-numpy-latest
test-conda-python-3.13
test-conda-python-3.13-pandas-nightly-numpy-nightly
test-conda-python-3.13-pandas-upstream_devel-numpy-nightly
test-conda-python-3.14
test-conda-python-emscripten
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-cuda-python-ubuntu-24.04-cuda-13.0.2
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-42-python-3
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-python-3

AlenkaF · 2025-12-12T09:29:57Z

Oh yes! Forgot about that, thanks for the info 👍

HyukjinKwon · 2025-12-12T22:48:38Z

@github-actions crossbow submit -g python

github-actions · 2025-12-12T22:51:31Z

Revision: 1a13b48

Submitted crossbow builds: ursacomputing/crossbow @ actions-12f77fab36

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-1.3.4-numpy-1.21.2
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.12-pandas-latest-numpy-1.26
test-conda-python-3.12-pandas-latest-numpy-latest
test-conda-python-3.13
test-conda-python-3.13-pandas-nightly-numpy-nightly
test-conda-python-3.13-pandas-upstream_devel-numpy-nightly
test-conda-python-3.14
test-conda-python-emscripten
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-cuda-python-ubuntu-24.04-cuda-13.0.2
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-42-python-3
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-python-3

HyukjinKwon · 2025-12-12T22:56:46Z

Rebased to retrigger the tests. #48449 (comment) is still valid.

test-conda-python-3.11-hypothesis passed

HyukjinKwon · 2025-12-12T23:05:55Z

Seems like:

tests/test_extension_type.py .................                           [ 40%]
Fatal Python error: Segmentation fault

Current thread 0x0000000203059040 (most recent call first):
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pyarrow/tests/test_fs.py", line 1224 in test_s3_options
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/python.py", line 166 in pytest_pyfunc_call
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/python.py", line 1720 in runtest
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/runner.py", line 179 in pytest_runtest_call
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/runner.py", line 245 in <lambda>
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/runner.py", line 353 in from_call
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/runner.py", line 244 in call_and_report
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/runner.py", line 137 in runtestprotocol
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/runner.py", line 118 in pytest_runtest_protocol
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/main.py", line 396 in pytest_runtestloop
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/main.py", line 372 in _main
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/main.py", line 318 in wrap_session
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/main.py", line 365 in pytest_cmdline_main
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/config/__init__.py", line 199 in main
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/config/__init__.py", line 223 in console_main
  File "/Users/runner/hostedtoolcache/Python/3.11.9/arm64/bin/pytest", line 7 in <module>
tests/test_fs.py ....sssx.xsss....sssx.xssss

Failure at MacOS is globally happening. I retriggered but still the issue persists. Let me leave it as is for now - it won't be related to my change in any event.

AlenkaF · 2025-12-16T07:44:55Z

Yes, macOS failures are not related.
The cuda extended build seems strange - not related, I think.

Thanks for all the work @HyukjinKwon!
Will wait for another pair of 👀
@kszucs in case you have time for one quick strategies review.

HyukjinKwon requested review from AlenkaF, raulcd and rok as code owners December 11, 2025 03:58

HyukjinKwon commented Dec 11, 2025

View reviewed changes

github-actions bot added Component: Python awaiting review Awaiting review awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Dec 11, 2025

HyukjinKwon force-pushed the GH-48448 branch from 0f436c0 to de8e939 Compare December 11, 2025 04:02

AlenkaF approved these changes Dec 12, 2025

View reviewed changes

HyukjinKwon force-pushed the GH-48448 branch from de8e939 to 1a13b48 Compare December 12, 2025 22:48

Implement Alphanumeric and Surrogate text in Random schema generator

1e3016a

HyukjinKwon force-pushed the GH-48448 branch from 1a13b48 to 1e3016a Compare December 12, 2025 22:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

GH-48448: [Python] Implement Alphanumeric and Surrogate text in the random schema generator #48449

GH-48448: [Python] Implement Alphanumeric and Surrogate text in the random schema generator #48449

HyukjinKwon commented Dec 11, 2025 •

edited by github-actions bot

Loading

Uh oh!

HyukjinKwon Dec 11, 2025

Uh oh!

github-actions bot commented Dec 11, 2025

Uh oh!

AlenkaF commented Dec 12, 2025

Uh oh!

AlenkaF left a comment

Uh oh!

HyukjinKwon commented Dec 12, 2025

Uh oh!

github-actions bot commented Dec 12, 2025

Uh oh!

AlenkaF commented Dec 12, 2025

Uh oh!

HyukjinKwon commented Dec 12, 2025

Uh oh!

github-actions bot commented Dec 12, 2025

Uh oh!

HyukjinKwon commented Dec 12, 2025 •

edited

Loading

Uh oh!

HyukjinKwon commented Dec 12, 2025

Uh oh!

AlenkaF commented Dec 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

GH-48448: [Python] Implement Alphanumeric and Surrogate text in the random schema generator #48449

Are you sure you want to change the base?

GH-48448: [Python] Implement Alphanumeric and Surrogate text in the random schema generator #48449

Conversation

HyukjinKwon commented Dec 11, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

HyukjinKwon Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Dec 11, 2025

Uh oh!

AlenkaF commented Dec 12, 2025

Uh oh!

AlenkaF left a comment

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Dec 12, 2025

Uh oh!

github-actions bot commented Dec 12, 2025

Uh oh!

AlenkaF commented Dec 12, 2025

Uh oh!

HyukjinKwon commented Dec 12, 2025

Uh oh!

github-actions bot commented Dec 12, 2025

Uh oh!

HyukjinKwon commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon commented Dec 12, 2025

Uh oh!

AlenkaF commented Dec 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

HyukjinKwon commented Dec 11, 2025 •

edited by github-actions bot

Loading

HyukjinKwon commented Dec 12, 2025 •

edited

Loading