_PyExecutorObject is exposed to the abidiff tool due to indirect use in PyCodeObject

[Fidget-Spinner]

Bug report

Bug description:

The problem is that PyCodeObject is thought of as part of the stable ABI in abidiff. _PyExecutorArray and thus _PyExecutorObject is not an opaque struct, so its fields and offsets are exposed as part of the stable ABI seen by the tool. Theoretically, this should be fine and not lead to crashes long as no one accesses the _PyExecutorObject, but still this is pretty annoying to have if we are to backport changes to the JIT that require changing the executor, as this will trigger a warning on abidiff tools.

There are two possible fixes, turn the executor object in the public header to be an opaque struct, or use a void pointer. I think both are fine, I'm inclined to link to the first one for type-safety reasons).

We can only fix this on main, as any prior version would break the ABIdiff/dump tools.

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

[ZeroIntensity]

We can only fix this on main, as any prior version would break the ABIdiff/dump tools.

Hm, won't this break limited API users? If you compile for the stable ABI on 3.14, it should work on 3.15 (and 3.16, and 3.17, and so on) as well. There's basically no way to remove something from the stable ABI once people have compiled for it. I think your best option here is to put a bunch of padding where _PyExecutorObject currently is relative to PyCodeObject (to avoid changing the size of the struct), and then move the actual field elsewhere.

Or, did you mean that _PyExecutorObject is exposed to the public (not limited/stable) C API? The documentation doesn't show PyCodeObject as being part of the stable ABI.

[kumaraditya303]

The problem is that PyCodeObject is part of the stable ABI.

No, it is not part of stable ABI. Only things listed in https://github.com/python/cpython/blob/main/Misc/stable_abi.toml are part of stable ABI.

[Fidget-Spinner]

Sorry I wasn't clear, I should've made it more explicit that the problem is that it's exposed to the abidiff tool through PyCode_AddWatcher.

[encukou]

Please don't use the term "Stable ABI" here :)

[Fidget-Spinner]

Please don't use the term "Stable ABI" here :)

I've changed all mentions in the original issue to mention the abidiff tool instead. Is that good enough?

[encukou]

Yes, thank you!

turn the executor object in the public header to be an opaque struct

That's the way to go. It'll also help ensure that the struct isn't used in public macros (of course if, it is used there, you can't change things in backports).

[kumaraditya303]

There are two possible fixes, turn the executor object in the public header to be an opaque struct, or use a void pointer. I think both are fine, I'm inclined to link to the first one for type-safety reasons).

Changing the executor to a opaque struct seems good to me.

We can only fix this on main, as any prior version would break the ABIdiff/dump tools.

You can do it in 3.14 as well as the struct is private, it will just require regenerating the abi file which is harmless in this case.