[SPARK-16948][SQL] Use metastore schema instead of inferring schema for ORC in HiveMetastoreCatalog #14537
…tion
- Loading branch information
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -238,7 +238,13 @@ private[hive] class HiveMetastoreCatalog(sparkSession: SparkSession) extends Log | |
| partitionSpec) | ||
|
|
||
| val schema = fileType match { | ||
| case "parquet" => | ||
| val inferredSchema = | ||
| defaultSource.inferSchema(sparkSession, options, fileCatalog.allFiles()) | ||
| inferredSchema.map { inferred => | ||
| ParquetFileFormat.mergeMetastoreParquetSchema(metastoreSchema, inferred) | ||
| }.getOrElse(metastoreSchema) | ||
| case "orc" => | ||
| metastoreSchema | ||
|
Contributor
There was a problem hiding this comment. I went through the code path again, seems we must infer the schema here. In metastore, we store the table schema and partition columns.
Contributor
There was a problem hiding this comment. @cloud-fan As we discussed offline yesterday, this is probably fine since ORC supports column pruning. Therefore, when reading an ORC file in a partitioned table, the reader always ignores partition columns stored inside the physical file and uses the value encoded in partition directory path.
Contributor
There was a problem hiding this comment. We already have a test case for this case here. |
||
| case _ => | ||
| throw new RuntimeException(s"Cannot convert a $fileType to a data source table") | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm a little worried here. If the table is partitioned,
metastoreSchemawill always contain partition columns, and thus the merged schema will contain partition columns too. This means, we always read parquet files with partition columns, I think we may have a hidden bug.