feat: add column mapping support when reading tables #3954
Conversation
JonatanMartens
requested review from
hntd187,
ion-elgreco,
roeap and
rtyler
as code owners
github-actions
bot
added
binding/python
|
ACTION NEEDED delta-rs follows the Conventional Commits specification for release automation. The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. |
JonatanMartens
changed the title
|
|
||
| if ( | ||
| table_protocol.reader_features | ||
| and "columnMapping" in table_protocol.reader_features | ||
| ): | ||
| raise DeltaProtocolError( | ||
| "The table requires reader feature 'columnMapping' " | ||
| "but this is not supported using pyarrow Datasets." | ||
| ) | ||
|
|
||
| if ( |
There was a problem hiding this comment.
Why is this removed? The pr itself doesn't provide support for reading column mapping with pyarrow
There was a problem hiding this comment.
My primary use case is loading delta tables with datafusion (in python not rust), so must have missed it.
Will look into adding support for reading with pyarrow
There was a problem hiding this comment.
I have opened a pr to arrow apache/arrow#48289, once that is merged we can support column mapping in datasets.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3954 +/- ##
==========================================
- Coverage 26.24% 26.13% -0.12%
==========================================
Files 124 124
Lines 19824 19914 +90
Branches 19824 19914 +90
==========================================
+ Hits 5203 5204 +1
- Misses 14251 14340 +89
Partials 370 370 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
2 participants
Description
This pr adds the ability to read tables with the column mapping feature enabled.
Related Issue(s)
Partially implements #930, but lacks the ability to write to tables with column mapping.
Documentation