Python SDK incompatible with Python 3.14+ due to Pydantic V1 compatibility layer

[nopcoder]

Description

The Python SDK (lakefs-sdk) emits a warning when used with Python 3.14 due to incompatibility between the Pydantic V1 compatibility layer and Python 3.14. While the import currently succeeds, the warning indicates that core Pydantic V1 functionality may not work correctly.

Warning Message

/Users/barak/work/lakeFS/.venv/lib/python3.14/site-packages/lakefs_sdk/api/actions_api.py:21: UserWarning: Core Pydantic V1 functionality isn't compatible with Python 3.14 or greater.
  from pydantic.v1 import validate_arguments, ValidationError

Root Cause

The SDK is generated using OpenAPI Generator v7.0.1 (treeverse/openapi-generator-cli:v7.0.1.4), which produces Pydantic V1 code. A post-processing script (clients/python-static/pydantic.sh) wraps all pydantic imports with a try/except pattern:

try:
    from pydantic.v1 import validate_arguments, ValidationError
except ImportError:
    from pydantic import validate_arguments, ValidationError

This was intended to support both Pydantic V1 and V2 users. However, pydantic.v1 (the compatibility shim in Pydantic V2) relies on Python internals that were removed in Python 3.14, making it potentially unreliable.

Affected features include:

• validate_arguments decorator (used in all 19 API files)
• V1-style constrained types (conint, constr, conlist)
• V1-style validators (@validator)

Environment

• Python version: 3.14
• lakefs-sdk version: current
• Pydantic version: 2.x (with degraded pydantic.v1 shim on Python 3.14)

Proposed Solution

1. Upgrade OpenAPI Generator from v7.0.1 to v7.2.0+

   • Starting with v7.1.0, the default python generator produces native Pydantic V2 code
   • A separate python-pydantic-v1 generator exists for backwards compatibility

2. Remove the pydantic.sh post-processing script

   • No longer needed when generating native Pydantic V2 code

3. Update dependency constraints

   • Change pydantic = "^1.10.5, <2" to pydantic >= 2.0 in pyproject.toml
   • Update setup.py accordingly

Files to Modify

• Makefile - Update OPENAPI_GENERATOR_IMAGE and remove PYTHON_POST_PROCESS_FILE
• clients/python-static/pydantic.sh - Remove or archive
• clients/python/pyproject.toml - Will be regenerated with correct pydantic constraint
• clients/python/setup.py - Will be regenerated with correct pydantic constraint

Impact

• Breaking change for users on Pydantic V1 (they would need to upgrade to Pydantic V2)
• Enables support for Python 3.14+
• Aligns SDK with modern Pydantic ecosystem

References

• [BUG] Python-pydantic-v1 is in the generator list, but python-pydantic-v1 is not in openapi-generator-cli 7.0.1 OpenAPITools/openapi-generator#16888 (python-pydantic-v1 availability)
• [PYTHON] update to pydantic v2 OpenAPITools/openapi-generator#16468 (pydantic v2 migration request)

[N-o-Z]

I also agree that the best solution here will be to upgrade the openapi generator version.
However I'd like to provide a more detailed analysis on this solution which also include additional considerations and implications on users:

Breaking Changes API:

Pydantic v1 (Current)	Pydantic v2 (After Upgrade)
.dict()	.model_dump()
.json()	.model_dump_json()
.parse_obj()	.model_validate()
.construct()	.model_construct()
class Config:	model_config = ConfigDict(...)
@validator	@field_validator
allow_population_by_field_name	populate_by_name
orm_mode	from_attributes
Optional[T] defaults to None	Optional[T] is required, allows None

---

Breaking Changes Async Users

Changes introduced:

1. async_req parameter removed - No longer exists
2. All methods marked async - True coroutines
3. Thread pool eliminated - No more ThreadPool
4. Direct awaitables - Methods return coroutines

Aspect	Current (Pre-7.1.0)	New (7.1.0+)
Async mechanism	ThreadPool	Native async/await
Parameter	async_req=True	N/A (removed)
Return type	AsyncResult	Coroutine
Result retrieval	thread.get()	await result
Code style	Can mix sync/async	Must be in async context

---

Impact on Users:

• Direct SDK users calling model methods (.dict(), .json(), etc.) must update code
• High-level wrapper (lakefs package) would need updates to use new method names
• Any user code that subclasses or extends SDK models needs migration
• For users using async_req=True:
  - Breaking change - Must refactor to true async/await
  - Code using .get() must change to await
  - Must wrap calls in async functions
  - May need asyncio.run() at top level

Pros:

• Fully solves Python 3.14+ compatibility
• Modern, non-deprecated code
• Better performance (Pydantic v2 is faster)
• Long-term sustainable solution
• Aligns with Python ecosystem direction

Cons:

• Breaking changes for existing SDK users
• Requires updating the high-level lakefs wrapper package
• Users must have Pydantic v2 (breaks v1 users)
• More testing required

[N-o-Z]

@ozkatz Need product input here.
In the past we supported both v1 and v2 relying on pydantic v2 compatibility layer with v1.
Now we want to remove the use of v1 altogether (including the use of the compatibility layer).
The way to do this is by upgrading the openapi-generator version which introduces significant breaking changes in the autogenerated SDK.
As long as the SDKs and lakeFS version is coupled - I don't believe we can introduce this change without a major release.

Another option will be to maintain a separate version of the autogenerated SDK (using a newer version of the openapi-generator) that will be used only by the HL Python SDK.
Doing that we can mitigate the breaking changes for user who use the autogenerated SDK explicitly, However:

1. This will still break users who use the underlying SDK via the HL Python SDK.
2. This will not solve the v1 warning for users who using the autogenerated SDK directly with a python 3.14 env.
3. Will require maintaining 2 versions of the internal SDK to some degree (we will not be publishing the new SDK)

[N-o-Z]

@nopcoder looking into the release documentation of pydantic v2.12 it states:

Pydantic V1 core functionality will not work properly with Python 3.14 or greater. As such, Python 3.13 is the latest supported Python version for V1.

Though they are not mentioning any "blocking" or removal of the compatibility layer for any future release, we cannot assume it is working properly even now.

I think we should move forward with creating an additional SDK (yet another one) and work towards sunsetting the current autogenerated SDK

[nopcoder]

thanks @N-o-Z . so let's mark this discovery and complete to plan this change as it is not something small.

[itaiad200]

@N-o-Z can we close this discovery item?

[N-o-Z]

@N-o-Z can we close this discovery item?

Can we close it even though we still don't have an agreed upon proposal?

[skshetry]

It looks like pydantic==1.10.25 has recently added support for 3.14. This support has also been backported to the pydantic.v1 compatibility API in v2, see:

• Add minimal support for Python 3.14 pydantic/pydantic#12636

A stable v2 release has not yet been published (maybe soon?), but two beta versions are available (2.13.0b1 and 2.13.0b2).

I tested 1.10.26 as well as 2.13.0b2, and the test suite passes in both cases. See #10201. There was a single test failure in LenientNamedTuple, caused by changes in annotation (see __annotations__ and PEP 649), but it was straightforward to fix.

That said, I have no idea how good our test coverage is. In any case, I think we should move to the v2 APIs.

[N-o-Z]

It looks like pydantic==1.10.25 has recently added support for 3.14. This support has also been backported to the pydantic.v1 compatibility API in v2, see:

• Add minimal support for Python 3.14 pydantic/pydantic#12636

A stable v2 release has not yet been published (maybe soon?), but two beta versions are available (2.13.0b1 and 2.13.0b2).

I tested 1.10.26 as well as 2.13.0b2, and the test suite passes in both cases. See #10201. There was a single test failure in LenientNamedTuple, caused by changes in annotation (see __annotations__ and PEP 649), but it was straightforward to fix.

That said, I have no idea how good our test coverage is. In any case, I think we should move to the v2 APIs.

Thanks for finding this. The underlying issue perfectly encapsulates our issue (users that use pedantic v1 compatibility layer on v2).
We just need them to release a new v2 and add a dependency requirement for python>=3.14

@nopcoder suggest we move this design for now to rejected

[N-o-Z]

Will be fixed in pydantic v2.13