ArrayV2Metadata.to_dict() fails when passed numpy type directly

[TomNicholas]

Zarr version

v3.1.0

Numcodecs version

n/a

Python Version

3.12

Operating System

linux

Installation

uv pip

Description

ArrayV2Metadata can error on .to_dict(). I think the problem is actually that it accepts a np.dtype as an attribute without any kind of validation on that field, which then sits around incorrectly until the .to_dict() surfaces the problem.

I assume what's supposed to happen is that parse_data_type is meant to be called on the input, because when I do that then .to_dict() behaves as expected.

Steps to reproduce

In [1]: import numpy as np

In [2]: from zarr.core.metadata import ArrayV2Metadata

In [3]: ArrayV2Metadata(
   ...:             chunks=(10,),
   ...:             shape=(5,),
   ...:             dtype=np.dtype("int32"),
   ...:             order="C",
   ...:             fill_value=None,
   ...:         ).to_dict()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[3], line 7
      1 ArrayV2Metadata(
      2             chunks=(10,),
      3             shape=(5,),
      4             dtype=np.dtype("int32"),
      5             order="C",
      6             fill_value=None,
----> 7         ).to_dict()

File ~/Documents/Work/Code/VirtualiZarr/.venv/lib/python3.13/site-packages/zarr/core/metadata/v2.py:227, in ArrayV2Metadata.to_dict(self)
    224     zarray_dict["fill_value"] = fill_value
    226 # pull the "name" attribute out of the dtype spec returned by self.dtype.to_json
--> 227 zarray_dict["dtype"] = self.dtype.to_json(zarr_format=2)["name"]
    229 return zarray_dict

AttributeError: 'numpy.dtypes.Int32DType' object has no attribute 'to_json'

In [4]: from zarr.dtype import ZDType, parse_data_type

In [5]: ArrayV2Metadata(
   ...:             chunks=(10,),
   ...:             shape=(5,),
   ...:             dtype=parse_data_type(np.dtype("int32"), zarr_format=2),
   ...:             order="C",
   ...:             fill_value=None,
   ...:         ).to_dict()
Out[5]: 
{'shape': (5,),
 'chunks': (10,),
 'dtype': '<i4',
 'fill_value': None,
 'order': 'C',
 'filters': None,
 'dimension_separator': '.',
 'compressor': None,
 'attributes': {},
 'zarr_format': 2}

Additional output

No response

[d-v-b]

I would use the type annotations as a guide here. The dtype parameter in __init__ is clearly annotated as ZDType:

zarr-python/src/zarr/core/metadata/v2.py

Line 79 in 2420f9e

dtype: ZDType[TDType_co, TScalar_co],

the dtype attribute is also annotated as ZDType:

zarr-python/src/zarr/core/metadata/v2.py

Line 66 in 2420f9e

dtype: ZDType[TBaseDType, TBaseScalar]

That means passing something in that's not a ZDType (a numpy data type is not a ZDType) will not work.

The reason why we are strict about this is because there are multiple distinct zarr data types that use the exact same numpy data type -- we support VariableLengthUTF8 and VariableLengthBytes, and they both use np.dtype('object'). So a numpy data type does not in general specify the zarr data type.

[d-v-b]

we could definitely consider adding some runtime type checking (or even widening the type of the dtype parameter to accept things that are zdtype-like, such as strings). But then we are validating the same input twice, because we check data types higher up in the user-facing array creation routines

[TomNicholas]

Fair enough. I keep (wilfully) forgetting that ArrayV2/3Metadata is not public API, because it's so useful!

I would selfishly prefer adding additional runtime type checking, but really I guess static type checking should have caught this in VZ anyway. So feel free to close this.

[d-v-b]

It absolutely should be public api! We just need to ensure that it has the right design. I have a lot of ideas for this, but haven't composed a plan for it yet...

[TomNicholas]

If you make it public API then you definitely should add runtime type checking to this field, because confusing numpy dtypes with Zarr data types like I did is a very easy mistake to make, even if it is statically type-checkable.