Skip to content

Conversation

@alamb
Copy link
Contributor

@alamb alamb commented Dec 11, 2022

Draft as I have to sort out calling the correct calls in ListingTable but I think it is possible. Just putting up WIP in case anyone else is interested

Which issue does this PR close?

Closes #4580

Rationale for this change

See #4580

What changes are included in this PR?

  1. Add dynamic catalog wrappers in datafusion-cli that treat unknown table names as a filename

Are these changes tested?

Not yet

Are there any user-facing changes?

For DataFusio-cli

Example use

cat ~/Downloads/parquet_select.sql 
select * from '/Users/alamb/Software/arrow-datafusion/parquet-testing/data/alltypes_tiny_pages.parquet' limit 10;
cargo run -- --file ~/Downloads/parquet_select.sql 

Current output:

DataFusion CLI v15.0.0
Providing catalog for name datafusion
Providing schema for public
Providing schema for information_schema
Providing catalog for name datafusion
Providing schema for public
Getting provider for /Users/alamb/Software/arrow-datafusion/parquet-testing/data/alltypes_tiny_pages.parquet
Treating /Users/alamb/Software/arrow-datafusion/parquet-testing/data/alltypes_tiny_pages.parquet as filename...
Plan("table 'datafusion.public./Users/alamb/Software/arrow-datafusion/parquet-testing/data/alltypes_tiny_pages.parquet' not found")

@timvw
Copy link
Contributor

timvw commented Jan 7, 2023

Most code to figure out what kind of table that needs to be built exactly is probably written in ListingTableFactory already..

@alamb
Copy link
Contributor Author

alamb commented Jan 7, 2023

@unconsolable has a PR to add this feature -- I hope to review it tomorrow #4838

@alamb
Copy link
Contributor Author

alamb commented Jan 8, 2023

@unconsolable fixed it in #4580

@alamb alamb closed this Jan 8, 2023
@alamb alamb deleted the alamb/automatic_catalog_list branch August 8, 2023 20:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support select .. FROM 'parquet.file' in datafusion-cli

2 participants