Heterogeneous column types

[jblondin]

So, I thought I'd start opening up issues to enable discussion of individual dataframe features we'd like to see. I'd like to start with 'heterogeneous column types': the ability to have a dataframe with columns of different types.

In looking through existing WIPs in #4, I came across a few different methods of implementing this:

1. Using an enum for either a column or for individual values. utah (and really any arbitrarily-typed dataframe library) can house enums as values, which allows you to mix types however you want (even within the same column), at the cost of run-time type safety and some performance. I didn't see any library currently use column-based enums, but I could see having something like

enum Column { 
    Float(Vec<f64>),
    Int(Vec<i64>),
    Text(Vec<String>), 
}

and in fact did it this way in an early version of agnes.

2. Using Any-based storage, along with some metadata for relating columns to data types at run-time. Used by rust-dataframe and black-jack.
3. Using cons-lists to provide compile-time type safety. Used by agnes and frames.

Each of these has its own advantages and disadvantages. For example, 1 and 2 lack compile-time type-checking, but have much cleaner type signatures and potentially cleaner error messages than 3 (where you have something like DataFrame<Cons<usize, Cons<f64, Cons<String, Nil>>>> for a relatively simple three-column dataframe).

You could also have a combination of the above techniques -- I could see something like cons-list type-checking column metadata structure while the data itself is stored in some sort of Any-based structure.

I'm personally a fan of the compile-time type-checking that cons-lists provide, but they can be hard to work with for those unfamiliar with them. I've started work on a labeled cons-list library (which will replace the one I'm using in agnes) to hopefully help out with some of these issues.

What are everyone's thoughts / opinions? Are there other options we should consider? I'd love to hear what people think the best approach is!

[jesskfullwood]

If you don't have type-safety (option 3), then why use Rust? Pandas is good enough (actually I think data.table is the best dataframe library, shame about the rest of R).

On the point of bad type signatures, for frames I made some horrible macros to generate the types
Frame1<T>, Frame2<T1, T2>, Frame3<T1, T2, T3> up to Frame32<...> as aliases for Frame<Cons<T1, Cons<T2, Cons<...>>>>.

This doesn't help when Rust reports errors, but it does make it possible to describe a Frame in a relatively succinct way. Especially since you can then alias like

type MyFrame = Frame6<...>

(Of course as soon as you do a join or a groupby or whatever, the type changes)

[jblondin]

On the point of bad type signatures, for frames I made some horrible macros to generate the types
Frame1<T>, Frame2<T1, T2>, Frame3<T1, T2, T3> up to Frame32<...> as aliases for Frame<Cons<T1, Cons<T2, Cons<...>>>>.

That does seem pretty useful. Also, since macros can be used in type contexts, you could have something like:

type MyFrame = Frame![Col1, Col2, Col3];

I did something similar with creating label lists in agnes.

The type signatures in errors are still a problem, though.

[LukeMathWalker]

It looks like it's a bit of a trade-off between compile-time type-checking vs readability and usefulness of error messages.
I have seen you have made some progresses on lhlist @jblondin - what do you think of it? Does it provide a significant benefit over raw Cons lists?

[jblondin]

I'm pretty happy with how lhlist is coming along. It's intended to implement the 'extensible record' model from the Haskell HList paper (section 5), and it's getting fairly close: I've implemented creation, label-based implementation, and iteration. I need to add some set membership functionality (currently you can reuse a label in a list, but lookup will only access the first one), work on the error messages a bit, and add a little bit of useful functionality (e.g. concatenation).

I think it'll be useful for what we're doing in that it'll provide a framework for our goal of compile-time-checked non-stringy column indexing. It should make the column lookup easier to use and a lot less messy.

[DeppLearning]

(actually I think data.table is the best dataframe library, shame about the rest of R).

You are not alone! https://h2oai.github.io/db-benchmark/
FST, somewhat related, allows rapid serialization of data.tables to disk: https://github.com/fstpackage/fst
Maybe some of their tricks apply to rust, I'm too much of a beginner in Rust to judge, so.. sorry if this doesn't fit here.

[jesskfullwood]

We use data.table + fst a lot at day job. We batch-process a large amount of data in-memory on a single beefy server (RSS around 100GB) and are in this annoying sweet spot where data.table by far the fastest tool we have found. I say unfortunate because you really shouldn't be writing serious production code in R but anything else would compromise performance.

data.table itself is highly optimized C with over a decade of development behind it. Still, I reckon a decent pure-Rust version could beat it handily. In particular, data.table uses R routines for string handling which are still super slow. I benchmarked a simple Rust version as around 2X slower on numerical workloads (SIMD?) but 12X faster for strings.

[mamcx]

If you don't have type-safety (option 3), then why use Rust?

Because that type-safety is for the LANG, but data CAN'T BE type safe in a universal way. Imagine I open psql, and start typing queries. Is interactive, and it can be type safe all.

My opinion is that any library that need to operate on arbitrary data MUST be dynamic, you could add a type safe layer on top "easily" but not the other way.

[FelixBenning]

Is this discussion/project still alive?

[FelixBenning]

I think the issue with the third proposal is, that I still have not understood it. It seems complicated. What are Cons-lists? And regarding the discussion of enums in #3 :
While it makes a lot of sense at first that this

enum Sex { Male, Female, NotStated }

is better than

is_male: bool

You still have the issue that you need to handle these enums in your eventual calculations. I mean it is nice that your dataframe can store arbitrary types. But you do not just want to store them. This is not supposed to be a database. And once you try to implement simple functions you could apply to these columns, like mean, you already need some knowledge about the type. Or at least assumptions you can make.
And these assumptions could probably be encoded as a trait. And then you could have Columns be an enum of these traits.

I mean for enum types like the Sex enum proposed above, you would probably want a way the ability to convert it into a dummy variable. I.e. it would implement a trait like "dummyable", which would tell you how this enum would be converted into multiple columns of dummyvaraibles (which are of bool type).
e.g. Sex would be converted into

conversion	is_female	is_na
Male	0	0
Female	1	0
Not Stated	0	1

It might be possible to provide a default implementation for arbitrary enums, by making the first element of the enum be the default i.e. without a dummy variable. But in some cases you might want a different dummy conversion. E.g. in case of non-smoker, occasional smoker and smoker, you might want:

conversion	is_occ_smok	is_smoker
non-smoker	0	0
occasional smoker	1	0
smoker	1	1

Since (if you want to do a linear regression) the effects of regular smoking are added on top of the occasional smoking effects.
On the other hand you would want a different conversion, when you want to get proportions (i.e. when you want to take the mean).

Anyway the point is: if you want to enable basic statistics/machine learning modules to build on top of dataframes. Then they need to be able to make some assumptions about the dataset. So an enum of allowed types for the columns might not be an antipattern after all.

---

And if you want to use truly unique types, then you would have to reimplement all the functionality the dataframe provides for this particular type. Which at this point, defeats the purpose of using a dataframe framework. And you would not want to build this functionality on top of this dataframe framework, if the types are encoded as some strange Cons-List, which you would have to first understand before you can build anything on top.
Ideally you could extend the existing enums or have a default/custom element in the type enum which allows you to define functionality for these unique things.

But I think that it is unlikely that there are more types than:
integer, float, string, categorical.

[dexterlemmer]

[Edit: Added warning to skip some of my comments.]

WARNING: To any body who haven't yet suffered through the next several comments by me and @FelixBenning: I suggest you skip all of my comments starting from this one (inclusive) up to (and including) #6 (comment). The only comment in this mess that I feel may have a sufficiently high cost/benefit ratio for most readers is #6 (comment). Just watch out for @FelixBenning's comment in between. I personally don't think it's worth while reading it, but he might disagree. 😉

@FelixBenning Regarding cons lists, I think @jblondin meant HLists. A cons list is just a special type of linked list. An HList is a cons list implemented in the type system. This makes it possible to define and reason about arbitrary length heterogeneous lists of types at compile time. So for example you can statically check the type of the i'th element in HList![T1, T2, ..., Tn] for any n: usize and i: 0..n. The disadvantage is that HList isn't actually a type. It's a kind and since Rust doesn't yet support HKT's, the ergonomics of working with HLists are terrible.

Regarding enum Sex { Male, Female, NotStated } is better than is_male: bool:

1. Isn't this off-topic here? Why didn't you post it on Use-cases #3?
2. I'll answer you, anyway, because my answer gives further insight on what I think we are striving for figuring out how to implement with this issue (Heterogeneous column types #6): I've done what you did myself in Python. However it sucks compared to actually being able to have columns of any type. There are multiple approaches in Rust (such as approach 3 (HLists) of this thread) that allows you to have your cake and eat it too. i.e. The implementer of the dataframe crate can get total support for all types for free since the compiler knows exactly what type the data in every column is, the following code should work just fine (barring any mistakes I might've made, since I last programmed in Rust before epoch 2018 was out so my Rust is a bit... rusty):

import heterogeneous_dataframe::{DataFrame, DF};

// A type the crate user just made up so the imported crate cannot possibly know
// about it
trait Foo {
  fn foo(&Self);
}
struct Foo {}
impl Foo for Foo {
  fn foo(&Self) {println!("Foo");}
}

// Another type the imported crate knows and cares nothing about
#[Derive(Debug)]
enum Sex {Male, Female, NotStated}

// A nice macro for easily constructing syntax sugar. It also generates some
// Zero Sized Types (ZST) for indexing specific columns and constructs
// the HList that enables statically typed heterogeneously typed columns:
let df = DF!{
  let name = vec!["John", "Mary", "Anon"];
  let age = vec![33u8, 44, 0];
  let sex_col = vec![Sex::Male, Sex::Female, Sex::NotStated];
  let foo_col = vec![Foo{}, Foo{}, Foo{}];
}

// Compiles since the compiler knows the type of the `name` column is String.
assert_eq!(df[Name, 1], "Mary")

// Compiles and prints "Sex::NotStated" since the compiler knows the type of
// the `sex_col` column is `Sex`
println!("{df[SexCol, 2]:?}")

// Compiles and prints "Foo" three times since the compiler knows the type of
// the `foo_col` column is `Foo`
for x in df[FooCol] {
  x.foo()
}

// This throws a type error at compile time if un-commented,
// since the compiler knows that the type of the `sex_col` column is not `u32`:
//let myval:u32 = df[SexCol, 1];

[dexterlemmer]

Edit: You might want to skip this...

Oh. Another disadvantage of HLists is that since its type must be completely known at compile time, you have to specify the exact number and types of columns at compile time. Therefore, even though something like this might work

let df = DataFrame::read_csv!::<(f32, f32, f32, u32)>("/path/to/data.csv");

This won't work if we use HLists internally, since the read_csv macro cannot possibly construct the HList at compile time:

let df = DataFrame::read_csv!("/path/to/data.csv");

[dexterlemmer]

Edit: You might want to skip this...

A proposed fourth approach to heterogeneous column types

We seem to have a tradeoff: On the one hand it would be very nice if we can have compile time type checks and zero cost abstractions, but on the other hand we need the ability to construct dataframes based on data that might only be available at runtime. (Bonuses like fast and ergonomic indexing, slicing and iteration and consistently good error messages would also be great.)

Can we perhaps eliminate the trade-off (or at least give the user the option to choose)and yet have a single consistent type? We might be able to after all. Here's what I think:

Approach 4: A typed front-end for a byte-array (or usize-array) back-end

1. API: Column Categories: We define four categories of columns:

   1. C1<T:Sized> Statically typed T, Statically known n_cols: i.e. The amount of columns in category C1 is known at compile time and so is the type of each C1 column, the type of the column must implement Sized for the back-end to work. We store the columns' metadata (name, index, type) in an HList or on some other kind of heterogeneous type list based on generics and associative types.
   2. C2<ColumnEnum> Dynamically typed T, Statically known n_cols: We store the columns' metadata in an array of our own Enum (i.e. [Column]).
   3. C3<ColumnEnum> Dynamically typed T, Dynamically computed n_cols, but still with the advantages of using our own enum such as :Sized and stack allocated: We store the columns' metadata in a Vec<ColumnEnum>.
   4. C4<dyn T:Any> Dynamically typed T, Dynamically computed n_cols and pretty much any type allowed as long as it implements certain types we need like Copy, Clone and probably also Serialize/Deserialize, etc.: This would probably require an approach like black-jack's, which uses baggie internally to store its data. (Note that baggie itself won't work, since it's a HashMap, not some kind of array, but we can may be store a similarly designed array of Box<dyn Any>'s or bind them to dyn Any+Sized or something (I don't quite know how Any works, it's after my time) to at least make accessing and iterating over either the pointers to them or themselves fast.

2. API: DataFrame Types: We define four types of DataFrames for ensuring that users don't pay for what they don't use and that they can know and control which features the DataFrames don't have:

   1. DF1: Contains only C1 columns. Stores column metadata such as strides, pointers to starts of columns, column indexces, etc. in arrays for fast and ergonomic indexing. Supports indexing through ZST, or label:Hash+Eq+Sized+Copy, or index:usize.
   2. DF2: The same as DF1, except that it contains a mixture of C1 and C2 columns (in an arbitrary order, so you can have columns arranged like [C2, C1, C1, C2, C2] and not just like [C1, C1, C2, C2, C2].
   3. DF3: Adds C3 columns at the end of a DF2. Also replaces all metadata arrays (except for ZST indexces) with vecs.
   4. DF4: Adds C4 columns at the end of DF3.

3. API: Creating dataframes: We can obviously use macros. We can also use functions, despite the HLists if we use the builder pattern. (We have to construct an HList column by column with one function call per column because we need to recursively define associated types to compute the HList. After the HList, the rest of the construction can obviously easily be done in one function call for each category of columns.)

   • Actually, I think we will need macros. The reason for this is that the indexing ZST's will need to be uniquely defined and will need associated TypeNums for making it possible to compute pointers to cells (explained below).

4. Back-end: Just a giant vector of bytes (or may be usize's would make more sense). The real magic is in how we connect the API with the back-end, not in the back-end itself. That said, we'll probably need to make our own implementation to ensure for example an optimal resize strategy when adding capacity.

5. Implementing the Index traits for each type of index to get ergonomic yet fast access to a single cell.:

   1. First convert the index into an offset into the back-end array as fast as possible using the available metadata in stored in the dataframe. Note that ZST indexes' offsets should be computed during compile time.
   2. If a ZST was used, we can do an unsave memory access (with due diligence in keeping track of lifetimes) for indexing without bounds checks.
   3. If a label was used, we might still be capable of skipping bounds checks if we do use a Label new type and the Label type is designed to prove that the label is for this dataframe and cannot be shared with the dataframe while either is mutable.
      • Note that on C1 columns we might need to make the label a constant in order for it to even work (since otherwise we cannot compute the return type for the Index trait). Hmm. Can the return type be computed even then?
   • Note: We'll need disambiguation between usize labels and indexes, either by creating a Label newtype, storing the label in an Option or by using the Pandas approach of iloc and loc indexes (although I'm not certain loc and iloc can have the same ergonomic syntax in Rust as in Pandas)
   4. We cannot implement the Index trait for C1 columns for usize indexes since the return type cannot be computed at compile time. Otherwise these are normal.
   5. If the caller provides a const usize index, however, it'll work for all column categories.
      • Note: C3 and C4 columns will definitely need runtime bounds checking since we cannot know at compile time how many such columns there are.

6. Implementing index slices:

   1. This is done similarly to implementing single cell indexing. However, notice that a dataframe acts like a smart pointer (or a vec). We'll need to create a new dataframe (or a type with a name like DataFrameSlice) with new metadata, but if the user asked for a reference, we won't need to copy, clone or move the back-end array.

7. Implementing zero cost row-, column- and window iterators:

   1. A rows iterator (i.e. iterator iterating down columns, not across columns) would be often used and simple to design, implement and optimize. Notice that doing this across multiple columns or even multiple categories of columns doesn't make this any harder, so we also have our first window iterator.
   2. As long as a columns iterator (i.e. iterating across columns, not down them) is only iterating over either just C2/C3 columns or just C4 columns, it's again simple.
   3. To extend the columns iterator to iterate over both C2/3 columns and C4 columns simultaneously, one of three strategies can be followed:
      1. Coerce the C2/3 cells into the type of C4. Yuck! However it should work for most use cases.
      2. Return two iterators in a tuple. Provide an API that allows seamlessly switching over from the first to the second. I have a sort of idea how this can be done, but can't quite figure it out. Note if the iterator can roll over to the next row, it gets more interesting, but we can perhaps just use a single pair of iterators for all rows and ping pong work between them.
      3. Simply not implement the capability. 🤷‍♂️
   4. To extend the columns iterator to handle C1 at all (whether on its own or together with other column categories), one of the following three approaches will have to be taken:
      1. Coerce every C1 cell into the type of C4. (Yuck! And I don't know if it's even possible and useful.)
      2. For each row: Create multiple iterators, put them in a tuple and somehow integrate them. Note that this will have to happen at compile time. At first glance that means, we cannot wrap to the next row, but we probably can if we keep the entire tuple of iterators in memory and just return back to running the first iterator in the tuple after the last (obviously incrementing the row index first).
      3. Again, simply skip implementing the capability.
   5. For the special case of a columns iterator constructed by the builder pattern rather than from a single function call, seamless integration should be easy across arbitrary column categories.

8. Missing data?:
   I'm not going to solve this here. I'll merely note that the back-end can probably easily be extended with a bitmask.

PS: Apologies for walls of text.

Especially since my previous long comment was mostly just a somewhat off-topic reply and I don't know whether the idea presented in this comment is any good either. ;-)

[FelixBenning]

1. Isn't this off-topic here? Why didn't you post it on #3?

No, but I apparently did not manage to get my point across very well. I am not interested in this particular example, i.e. I am not interested into a certain use case. Which is why this does not belong into #3. I instead tried to make a more general point using this as an example. So I'll take another shot at it clarifying what I meant:

I am going to play dumb and claim there only 5 types of objects in rust:

• integers
• floats
• bools
• enums
• Strings

So we can simply create an enum of these 5 types and have covered all possible types. This is of course not quite true. So what are the issues with this statement?

1. Sizes (i8, i16, i32, i64, i128, i256, i512, f8,...), but since there are only finitely many sizes which are actually used, this is not a fundamental problem of this claim, and would still allow an enum
2. Structs Of course you can create arbitrary many more objects using Structs. But the thing is: if you have struct data entries in a dataframe, you are arguably dataframing wrong. Every Row is literally a list of all the properties of one "object"/measurement. If some property has sub properties, you should arguably flatten this into additional columns. If you really want to retain some hierarchical structure of properties, you can achieve that with multiindexes.

But multi indexes could simply be implemented by creating a multiIndexDataframe, which is simply a list of enums with the two options multiIndexDataframePtr and dataframePtr. This way you can have arbitrarily deep multi indexes.

So what is left is basically just 5 possible types (times possible sizes). Which is perfectly suited for an enum. The only thing this could not deal with is oversized types, if someone starts to implement them. But I very much doubt that anyone is really going to need larger than 512 bytes for any kind of numeric value. And for SIMD we only need to keep our enum up to date with the current chip generation, which is very little work.

The only thing which might possibly cause issues, is String lengths. But if someone tries to store books in dataframe cells, then they probably care so little about performance, that we can also store string pointers.

---

Another argument is practicability (the above should be completely sufficient, but I started with this argument, so I will try to clarify this a bit more in order to explain my past comments):

When people use a dataframe library, they want to use prebuilt methods and functions. They want to be able to use stuff like import_from_csv as you suggested, they want to use .mean(), they want to use .fit_least_squares_linear_regression(), etc.
You can not possibly provide these features, if you do not know the type of objects they are supposed to apply to. And I tried to illustrate the issue of implementing a .fit_least_squares_linear_regression() by arguing how difficult it would be to implement this for an arbitrary enum (Sex, Smoker, etc.).
For this reason you would actually need some dummyfication on enums. You could have a default implementation for this dummyfication, but in a lot of cases, you would have to override this implementation as explained above. Which would currently be a problem, since you can not reimplement traits. But since specialization is coming, this would eventually solve itself.
For this reason I would actually rewrite my original list of types into

• Integers
• Floats
• Bools
• Dummyfiable (extends enums, is of size usize, implements asDummy())
• Strings

cc @dexterlemmer

[dexterlemmer]

[Edit6: Added section impl4 is a superset of all of impl1/2/3]
[Edit5: Added more column conversions for impl4.]
[Edit4: A bunch of u8s should've been usizes]
[Edit3: Edited in a new paragraph into ### 3. We can (and IMHO should) do better than impl1 (df_crate::ColEnum) by using impl2 (Any+metadata). It's marked.]
[Edit2: Strike through text that isn't valid any more.]
[Edit: Fixed a typo.]

Sorry everyone. I was an ass. I wrote 3 very long, badly structured and badly thought through comments above[^Note to self @dexterlemmer never write stuff on the Internet when you're tired! You knew that already.]. I'll delete my previous posts and write below what I should have from the start.

Terminology note: Implementations.

The OP talks about three implementation strategies for dataframes with heterogeneous column types. After this section, I'll just refer to them as impl1/2/3. Here's a recap of the implementation strategies:

• Impl1: A vec[ColumnTypes] where enum ColumnTypes is defined in the dataframe crate, not in the user's crate.
• Impl2: Something like a vec<Series<dyn Any>> (but clearly not preciesely that).
• Impl3: HLists (OP said ConsLists) of vecs.

I'll also propose a new implementation (and API):

• Impl4: Column<T>, AnyCol<T>, Row<S>, a bunch of DF types mirroring std's Tuple, Array and Vec, and finally some private buffers.

Overview

1. Explanation of impl3.
2. I feel impl3 (HLists) alone is insufficient.
3. We can (and IMHO should) do better than implementation 1 by using implementation 2.
4. Proposal: Impl4: Column<T>, AnyCol<T>, Row<S>, a bunch of DF types mirroring std's Tuple, Array and Vec, and finally some private buffers.
5. I would be willing to try implementing impl4 myself, but...

1. Explanation of impl3 (HLists)

This is simply a way for us to fully statically type our heterogeneous columns. In rust, the frunk crate implements HLists. Static typing is great. Most of what makes Rust great is that it's statically typed by default. Sure, many people probably won't claim they like Rust because it's statically typed, but I can guarantee you that almost anything most people might say is great about Rust either requires static typing or interact with other great features in a way that requires static typing.

Here's an example of using HLists from frunk's README (note the compiler allows this to type check and would've raised a type error if the types of the elements didn't match):

// You can convert a tuple to an HList and vice-versa
let h2 = hlist![ 42f32, true, "hello" ];
let t: (f32, bool, &str) = h2.into();
assert_eq!(t, (42f32, true, "hello"));

The advantage of HLists over tuples is that they are lists, so we can do things that are impossible with tuples such as iterate over an HList, pushing and popping elements, etc.

2. I feel impl3 (HLists) alone is insufficient.

HLists' strength (full static typing) is also their weakness:

1. We cannot dynamically change the types, number or order of columns. Furthermore, we also cannot implement abstractions that internally requires dynamic columns. For example:

   // This will not compile unless the compiler can infer the type from other code
   let _ = dataframe::read_csv("/path/to/data.csv")
   // We need to do this in stead
   let _ = dataframe::read_csv::<usize, f32, f32, f32, u8>("/path/to/data.csv")
   

2. HList-based dataframes are un-ergonomic:
   1. Ugly type errors are common. The compiler might tell you

      HCons<bool, HCons<&str, HCons<Option<usize>, HNil>>> does not match HCons<bool, HCons<&str, HNil>>
      

      rather than simply

      (bool, &str, Option<usize) does not match (bool, &str>)
      

   2. You cannot implement any trait designed for runtime values rather than for static types. For example, the Index and Iterator traits are impossible to implement for dataframes based on impl1. In stead users will have to use macros written by us or the frunk crate to access their data.

3. We can (and IMHO should) do better than impl1 (df_crate::ColEnum) by using impl2 (Any+metadata)

My whole series of posts started in reaction to @FelixBenning who stated that the restriction of impl1 (dataframes based on impl1 supports only a specific limited set of types) isn't a problem since any data type can be encoded in dataframes with only a few types. He showed as an example that we can one-hot encode any enum with unit variants.

To some extend I agree with @FelixBenning , we can indeed encode any data type. We can for example also encode an array by splitting its elements across multiple columns. Encode a vec by using an additional dataframe that encodes the items of the vec as separate rows, etc. Heck, since this is Rust, we can probably encode any type's digital representation (or that of a pointer pointing to it) and (with a considerable amount of effort) teach the borrow checker how to ensure accessing the data is still save!

But just because we could, doesn't mean that we should! All encodings lose convenience and type safety. Most encodings also have significant performance and/or memory overhead -- quite an important consideration in data science!

Edit: I've said the following in one of my comments I've instructed people to skip:

Why would we arbitrarily not allow our user to use Column<NonStandardFloat1024>.median() if he wants to? And why should we arbitrarily force a user to wait until runtime before he gets an error informing him he accidentally tried to call Column<f32>.add::<Column<bool>() in a situation where he already knows the datatypes of all his columns at compile time?

I assert that while dataframes based on impl1 are useful in separate crates, it would be a shame if we decide to focus the community's efforts on impl1. And as I understand it, focusing the community's efforts is what rust-dataframe -- and therefore this issue thread -- is all about.

4. Proposal: Impl4: Column<T>, AnyCol<T>, Row<S>, a bunch of DF types mirroring std's Tuple, Array and Vec, and finally some private buffers.

Types

We provide two column types:

• Column<T> is statically typed. Conceptually, a Column<T> is a vec<T>. Note that T: Sized+Copy, not just Sized (unlike a vec) because otherwise we can't make it work inside a dataframe[^It is possible someone may come up with a way to get rid of the restriction that the type contained by a `Column<T>` must be `Copy`.].
• AnyCol is a subtype of Column<T> and is dynamically typed. An AnyCol hides its type from collections containing it. Internally, it probably uses the Any trait.

We provide a single row type:

• Row<S>. Note that the type of S depends on the specific concrete dataframe type (see below).

We provide four dataframe types:

• trait AbstractDF<S, R:Row<S>> defines functions- and implements methods shared by the other two dataframe types.
• ColsTuple<Tuple<Column<T1>, ..>> : AbstractDF<Row<Tuple<T1, ..>>> is conceptually like a tuple of columns ((Column<T1>, ..)). It is completely statically typed. If we cannot implement it (though I cannot see why not), we can provide a ColsTuple<HList<Column<T1>, ..>> in stead. A user can of course store an AnyCol in it, but it doesn't really make sense, so we won't bother explicitly providing support for AnyCols.
• ColsArray<T, C:Column<T>> : AbstractDF<Row<[T]>> is conceptually like an array of columns ([<Column<T>]). Note that it is not heterogeneous. However, we should explicitly provide special support for a ColArray, which acts like a heterogeneous array.
• ColsVec<T, C:Column<T>> : AbstractDF<Row<vec<T>>> is conceptually like a vec<Column<T>>. Note that it is not heterogeneous. However, we should explicitly provide special support for a ColsVec, which acts like a heterogeneous vec.

Finally, none of the above column-, row- or dataframe types actually store any data. They're merely the API's over two hidden data buffers:

• ColumBuffer is pretty much just an [u8]. NB! ColumnBuffer does not actually know what type it's data is, only the size of the type. Otherwise, we cannot make DFBuffer (see below) heterogeneous. Indeed, we might dispence with the name or simply make it an alias for [u8].
• DFBuffer<S, R:Row<S>> is just an [ColBuffer]. Indeed, again, we might not even bother with the type or simply make it an alias for [ColBuffer]. NB! DFBuffer should not implement row accessing, since different dataframe types would handle accessing rows differently.
• Note: There's no RowBuffer type. You can, however, have a &DFBuffer that precisely slices a single row in a DFBuffer. Which will be important later on.

Indexing and slicing

For a start, we can probably simply provide the first 6 index types given below. Eventually we may support multiple index types:

1. usize for normal array-like indexing.
2. Range<usize> for normal array-like slicing.
3. (Range, Range) for slicing a DF conveniently. (It's not just about not requiring multiple sets of brackets. It's also about being able to index into a row at all (see below).)
4. (usize, usize) for consistency given the first three index types.
5. (usize, Range) for consistency given the first four index types.
6. (Range, usize) for consistency given the first four index types.
7. const usize or TypeNum for normal array indexing (with compile time bounds checking when used to index a ColsTuple).
8. HLabel<H:Hash> for indexing by label, backed by a HashMap<HLabel, usize>.
9. OLabel<O:Ord> for indexing by label in a sorted by index label DF or Column.
10. ColIndex, RowIndex and DFIndex. These are like &usizes. However they borrow from a Column or DF and are typed in such a manner that no bounds checking is ever necessary. (The borrow checker will ensure that they can never refer to a different Column or DF than the one we're using them to index and that we can never have shared mutable state between them and the Column or DF we're accessing.)
11. Unique zero sized types (ZST's) for those who want statically typed labels. Note that these would only be available for a Row<Tuple<..>> and for a ColsTuple, but not for a Row<[T]>, ColsArray, Row<vec<T>> or ColsVec.

Indexing and slicing a Column<T> (and hence also an AnyCol) should be just like indexing into a vec from an API point of view. Internally, impl Index<T> for Column<T> should use std::mem::transmute to convert between T and &[u8] since they store their data in their internal ColumnBuffer, which stores only u8s.

Indexing into a DF, returns the column at that index. Note that we can always compute the position of the column in the buffer if we store stride information.

We'll also need to provide row indexing, with AbstractDF<S, R:Row<S>>::index::<(Range, usize)>() -> &R and AbstractDF<S, R:Row<S>>::rows()[usize] -> &R.

Slicing a DF, always returns a &DF. so for example df_tuple[..] simply return a &DFTuple, etc.

WARNING: There's a little gotcha here: Between how slicing of a DF works and the way that the Index trait works we must use a tuple index if we want a row. Since df_tuple[..] returns a &ColsTuple, trying df_tuple[..][i] returns a &&Column<Ti>, i.e. the i'th column (not row). That said, letting a DF return a &DF if it's sliced, is very important for ergonomic and safe windowing. It also allows future proofing. (For example making the buffer partitioned like in Dask.) It's also consistent with the way arrays and vecs works, iirc.

Iterators

We should definitely implement both column-iterators and row-iterators. Note that implementing Row<(T1, T2, ..., Tn)>::into_iter() would be impossible in general. We could provide macros, but I think we should just refer the user to frunk and/or suggest he convert into a Row<[T]>. To make using frunk easy for our users, we should also support Row<HList<..>> even though none of the proposed dataframe types uses it directly.

Column conversions

Edit: We should provide at least the following conversions between our columns and other collections:

• impl From<Column<T>> for &[T] and impl Form<&[T]> for Column<T>
• impl From<Column<T, D>> for ndarray::Array<T, (1,D)> and impl From<Column<T, D>> for ndarray::ArrayView<T, (1,D)> and `impl ndarray::Array<T, (1,D)> for Column<T, D>``.
  • Note that here we have a dimension type parameter in column that I didn't include anywhere else. I just didn't want to unnecessarily complicate the rest of the code and furthermore, for a start we don't need the dimension parameter. For a start, we can just convert to/from ndarray's with dynamic dimensions and not bother with static dimensions ourselves.
• Note that we may need to explicitly provide column conversions to/from AnyCol and ExternalCollection<T: impl NumericWhatever>. I'm not sure how well Any would work without that.
• Note that we'll need to use std::mem::transmute to implement any of these conversions.

We should provide From<Column<T>> for AnyCol and From<AnyCol> for Column<T> or something similar. Note, I don't think the above mentioned conversion traits are legal, so we should come up with a good alternative.

Row conversions

We should impl From<S> for Row<S> and impl From Row<S> for S for all four types of Rows (i.e., tuple, hlist!, array and vec).

Converting between a Row<[T]> and a Row<vec<T>> would be straight forward. So would converting between a Row<(T1, T2, ..., Tn)) and a Row<hlist!<T1, T2, ..., Tn>>.

We should also (if possible) provide the ability to convert between a Row<[T]> and a Row<(T1, T2, ..., Tn)>. The conversion would require wrapping converting between Ti and dyn Any or something. Note that this will require macros if at all possible.

impl4 is a superset of all of impl1/2/3

If anyone still think that impl1/2/3 still has some advantage, do note that impl1/2/3 are all available to our users as a direct consequence of impl4's design:

• To get impl1 without impl1's arbitrary type restrictions, the user can simply use ColsArray<UserPreferredEnum, Column<UserPreferredEnum>> or the same with ColsVec.
  • For example, to provide a dataframe with a very similar API to Arrow's, he can just use Column<arrow::datatypes::DataType>. It won't be the same, but if either we or Arrow want to implement conversion between our dataframes and Arrow's it'll work and furthermore, Arrow would very likely be able to refactor their dataframe datatype to be based upon ours (assuming we add partitioning support).
• To get impl2, the user can simply use AnyCol's inside either ColsArray or ColsVec. He might want label-based accessing. For that we could either implement the label-based indexing I've proposed we do eventually for ColsArray and ColsVec or we could perhaps add a ColsMap dataframe type.
• ColsTuple is impl3 (except that it uses tuples, rather than hlists) and we can easily add a very similar ColsHList datatype). ColsTuple also gets rid of some of the disadvantages of impl3 because tuples would be less unergonomic (or at least less unintuitive to most users) and it can easily be converted into either a ColsArray<AnyCol> or a ColsVec<AnyCol> with zero cost whenever he needs to.

I would be willing to try implementing impl4 myself, but...

But it would be in my free time and I'll first need to re-acquaint myself with Rust first, because I've not really worked with Rust since about two years ago. Therefore, it'll probably take several weeks before I can even make a start. Therefore if someone else agree with me that we should go for impl4 and decides to run with it himself, that might be better.

[dexterlemmer]

Edit: You might want to skip this

OK. Since @FelixBenning has replied to one of my previous comments while I wrote the above one, I'll rather not delete my previous comments.

@FelixBenning. I'll read your reply and get back to you.