Skip to content

feat: Support FixedSizedListArray for length kernel #4520

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tustvold merged 6 commits into from
Jul 19, 2023
Merged

feat: Support FixedSizedListArray for length kernel #4520

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a9fbda4
feat: Support FixedSizedListArray for length kernel
Weijun-H Jul 13, 2023
b80afa0
fix clippy
Weijun-H Jul 13, 2023
07e99ea
update comment
Weijun-H Jul 16, 2023
aaf29c1
avoid unsafe
Weijun-H Jul 16, 2023
4c56e86
reduce useless trait
Weijun-H Jul 16, 2023
e76f7f5
remove T
Weijun-H Jul 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
remove T
  • Loading branch information
@Weijun-H
Weijun-H committed Jul 19, 2023
commit e76f7f5e0f6666045b933e413b09de2db7ed3b06
20 changes: 6 additions & 14 deletions arrow-string/src/length.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@

//! Defines kernel for length of string arrays and binary arrays

use arrow_array::types::*;
use arrow_array::*;
use arrow_array::{cast::AsArray, types::*};
use arrow_buffer::Buffer;
use arrow_data::ArrayData;
use arrow_schema::{ArrowError, DataType};
Expand Down Expand Up @@ -88,18 +88,12 @@ where
unary_offsets!(array, T::DATA_TYPE, |x| x)
}

fn length_list_fixed_size<T>(array: &dyn Array, length: i32) -> ArrayRef
where
T: ArrowPrimitiveType,
{
let array = array.as_any().downcast_ref::<FixedSizeListArray>().unwrap();
fn length_list_fixed_size(array: &dyn Array, length: i32) -> ArrayRef {
let array = array.as_fixed_size_list();
let length_list = array.len();
let buffer = Buffer::from_vec(vec![length; length_list]);
Copy link
Contributor

@tustvold tustvold Jul 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This assumes T::Native is i32


let data: PrimitiveArray<T> =
PrimitiveArray::new(buffer.into(), array.nulls().cloned());

make_array(data.into())
let data = Int32Array::new(buffer.into(), array.nulls().cloned());
Arc::new(data)
}

fn length_binary<O, T>(array: &dyn Array) -> ArrayRef
Expand Down Expand Up @@ -186,9 +180,7 @@ pub fn length(array: &dyn Array) -> Result<ArrayRef, ArrowError> {
DataType::LargeUtf8 => Ok(length_string::<i64, Int64Type>(array)),
DataType::Binary => Ok(length_binary::<i32, Int32Type>(array)),
DataType::LargeBinary => Ok(length_binary::<i64, Int64Type>(array)),
DataType::FixedSizeList(_, len) => {
Ok(length_list_fixed_size::<Int32Type>(array, *len))
}
DataType::FixedSizeList(_, len) => Ok(length_list_fixed_size(array, *len)),
other => Err(ArrowError::ComputeError(format!(
"length not supported for {other:?}"
))),
Expand Down