Skip to content

[datasets sql] Run the DuckDB CLI subprocess with UTF-8 encoding - #5020

Merged
Wauplin merged 2 commits into
huggingface:mainfrom
825pranav:fix-duckdb-cli-utf8
Sep 28, 2026
Merged

Wauplin merged 2 commits into
huggingface:mainfrom
825pranav:fix-duckdb-cli-utf8

Conversation

@825pranav

@825pranav 825pranav commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #5019.

_DuckDBCliRelation.execute runs the DuckDB CLI with text=True but no encoding, so the pipes use the locale encoding. DuckDB's stdin/stdout are UTF-8 on every platform, which on stock Windows (cp1252) means:

  • non-ASCII cells come back mojibake'd: SELECT chr(233) AS city → [{'city': 'é'}]
  • output bytes undefined in cp1252 kill the pipe reader (UnicodeDecodeError), result.stdout is None, and the library crashes with AttributeError: 'NoneType' object has no attribute 'strip'
  • non-ASCII in the query itself fails in _stdin_write with UnicodeEncodeError

One-line fix: pass encoding="utf-8" to the subprocess.run call. Verified on Windows 11 / cp1252 / DuckDB CLI v1.5.5 — all three cases return correct data (é, Á, 中文) after the change. The duckdb Python package path was never affected.

Added TestDuckDBCliRelationEncoding in tests/test_cli.py: mocks the subprocess and asserts the UTF-8 encoding kwarg; fails on main, passes with the fix. ruff check / ruff format --check clean; pytest tests/test_cli.py -m "not production" → 372 passed (the 2 failures also fail on a clean checkout of main in this environment).


Note

Low Risk
Single-parameter change aligning subprocess text mode with DuckDB’s documented UTF-8 behavior; scope is limited to the CLI dataset SQL path.

Overview
Fixes incorrect decoding when _DuckDBCliRelation.execute talks to the DuckDB CLI on locales where the default pipe encoding is not UTF-8 (e.g. Windows cp1252).

The subprocess.run call now passes encoding="utf-8" alongside text=True, so stdin/stdout match DuckDB’s UTF-8 I/O. That avoids mojibake on non-ASCII result cells, UnicodeDecodeError / empty stdout on some byte sequences, and UnicodeEncodeError when the SQL string contains non-ASCII text.

Reviewed by Cursor Bugbot for commit 243947d. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread tests/test_cli.py Resolved
Comment thread tests/test_cli.py Resolved
Co-authored-by: Lucain <lucainp@gmail.com>

@Wauplin Wauplin left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. Thanks for reporting and fixing 🤗

@Wauplin
Wauplin merged commit a076863 into huggingface:main Sep 28, 2026
12 of 21 checks passed
@bot-ci-comment

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@huggingface-hub-bot

Copy link
Copy Markdown
Contributor

This PR has been shipped as part of the v2.1.0 release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hf datasets sql returns mojibake or crashes on non-ASCII data when using the DuckDB CLI binary

2 participants