[SQL] SPARK-6489: Optimize lateral view with explode to not unnecessary columns.

[dreamquster]

No description provided.

[dreamquster]

What's wrong?Who can verify it?

[marmbrus]

ok to test

[SparkQA]

Test build #29754 has finished for PR 5358 at commit 376d332.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[chenghao-intel]

Is that OK if we just do like

case gen @ Generate(_, _, _, _, child) 
  if (child.outputSet -- gen.references).nonEmpty =>
  gen.copy(child = Project(a.references.toSeq, child))

[chenghao-intel]

Then we don't need other code change except the unit test.

[dreamquster]

Ok,thanks:)

[SparkQA]

Test build #29844 has finished for PR 5358 at commit 644f688.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[marmbrus]

This mutable state is no longer needed.

[SparkQA]

Test build #30300 has finished for PR 5358 at commit 9e7aaec.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[SparkQA]

Test build #30311 has finished for PR 5358 at commit 8909a5d.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch removes the following dependencies:
  • RoaringBitmap-0.4.5.jar
  • activation-1.1.jar
  • akka-actor_2.10-2.3.4-spark.jar
  • akka-remote_2.10-2.3.4-spark.jar
  • akka-slf4j_2.10-2.3.4-spark.jar
  • aopalliance-1.0.jar
  • arpack_combined_all-0.1.jar
  • avro-1.7.7.jar
  • breeze-macros_2.10-0.11.2.jar
  • breeze_2.10-0.11.2.jar
  • chill-java-0.5.0.jar
  • chill_2.10-0.5.0.jar
  • commons-beanutils-1.7.0.jar
  • commons-beanutils-core-1.8.0.jar
  • commons-cli-1.2.jar
  • commons-codec-1.10.jar
  • commons-collections-3.2.1.jar
  • commons-compress-1.4.1.jar
  • commons-configuration-1.6.jar
  • commons-digester-1.8.jar
  • commons-httpclient-3.1.jar
  • commons-io-2.1.jar
  • commons-lang-2.5.jar
  • commons-lang3-3.3.2.jar
  • commons-math-2.1.jar
  • commons-math3-3.1.1.jar
  • commons-net-2.2.jar
  • compress-lzf-1.0.0.jar
  • config-1.2.1.jar
  • core-1.1.2.jar
  • curator-client-2.4.0.jar
  • curator-framework-2.4.0.jar
  • curator-recipes-2.4.0.jar
  • gmbal-api-only-3.0.0-b023.jar
  • grizzly-framework-2.1.2.jar
  • grizzly-http-2.1.2.jar
  • grizzly-http-server-2.1.2.jar
  • grizzly-http-servlet-2.1.2.jar
  • grizzly-rcm-2.1.2.jar
  • groovy-all-2.3.7.jar
  • guava-14.0.1.jar
  • guice-3.0.jar
  • hadoop-annotations-2.2.0.jar
  • hadoop-auth-2.2.0.jar
  • hadoop-client-2.2.0.jar
  • hadoop-common-2.2.0.jar
  • hadoop-hdfs-2.2.0.jar
  • hadoop-mapreduce-client-app-2.2.0.jar
  • hadoop-mapreduce-client-common-2.2.0.jar
  • hadoop-mapreduce-client-core-2.2.0.jar
  • hadoop-mapreduce-client-jobclient-2.2.0.jar
  • hadoop-mapreduce-client-shuffle-2.2.0.jar
  • hadoop-yarn-api-2.2.0.jar
  • hadoop-yarn-client-2.2.0.jar
  • hadoop-yarn-common-2.2.0.jar
  • hadoop-yarn-server-common-2.2.0.jar
  • ivy-2.4.0.jar
  • jackson-annotations-2.4.0.jar
  • jackson-core-2.4.4.jar
  • jackson-core-asl-1.8.8.jar
  • jackson-databind-2.4.4.jar
  • jackson-jaxrs-1.8.8.jar
  • jackson-mapper-asl-1.8.8.jar
  • jackson-module-scala_2.10-2.4.4.jar
  • jackson-xc-1.8.8.jar
  • jansi-1.4.jar
  • javax.inject-1.jar
  • javax.servlet-3.0.0.v201112011016.jar
  • javax.servlet-3.1.jar
  • javax.servlet-api-3.0.1.jar
  • jaxb-api-2.2.2.jar
  • jaxb-impl-2.2.3-1.jar
  • jcl-over-slf4j-1.7.10.jar
  • jersey-client-1.9.jar
  • jersey-core-1.9.jar
  • jersey-grizzly2-1.9.jar
  • jersey-guice-1.9.jar
  • jersey-json-1.9.jar
  • jersey-server-1.9.jar
  • jersey-test-framework-core-1.9.jar
  • jersey-test-framework-grizzly2-1.9.jar
  • jets3t-0.7.1.jar
  • jettison-1.1.jar
  • jetty-util-6.1.26.jar
  • jline-0.9.94.jar
  • jline-2.10.4.jar
  • jodd-core-3.6.3.jar
  • json4s-ast_2.10-3.2.10.jar
  • json4s-core_2.10-3.2.10.jar
  • json4s-jackson_2.10-3.2.10.jar
  • jsr305-1.3.9.jar
  • jtransforms-2.4.0.jar
  • jul-to-slf4j-1.7.10.jar
  • kryo-2.21.jar
  • log4j-1.2.17.jar
  • lz4-1.2.0.jar
  • management-api-3.0.0-b012.jar
  • mesos-0.21.0-shaded-protobuf.jar
  • metrics-core-3.1.0.jar
  • metrics-graphite-3.1.0.jar
  • metrics-json-3.1.0.jar
  • metrics-jvm-3.1.0.jar
  • minlog-1.2.jar
  • netty-3.8.0.Final.jar
  • netty-all-4.0.23.Final.jar
  • objenesis-1.2.jar
  • opencsv-2.3.jar
  • oro-2.0.8.jar
  • paranamer-2.6.jar
  • parquet-column-1.6.0rc3.jar
  • parquet-common-1.6.0rc3.jar
  • parquet-encoding-1.6.0rc3.jar
  • parquet-format-2.2.0-rc1.jar
  • parquet-generator-1.6.0rc3.jar
  • parquet-hadoop-1.6.0rc3.jar
  • parquet-jackson-1.6.0rc3.jar
  • protobuf-java-2.4.1.jar
  • protobuf-java-2.5.0-spark.jar
  • py4j-0.8.2.1.jar
  • pyrolite-2.0.1.jar
  • quasiquotes_2.10-2.0.1.jar
  • reflectasm-1.07-shaded.jar
  • scala-compiler-2.10.4.jar
  • scala-library-2.10.4.jar
  • scala-reflect-2.10.4.jar
  • scalap-2.10.4.jar
  • scalatest_2.10-2.2.1.jar
  • slf4j-api-1.7.10.jar
  • slf4j-log4j12-1.7.10.jar
  • snappy-java-1.1.1.6.jar
  • spark-bagel_2.10-1.4.0-SNAPSHOT.jar
  • spark-catalyst_2.10-1.4.0-SNAPSHOT.jar
  • spark-core_2.10-1.4.0-SNAPSHOT.jar
  • spark-graphx_2.10-1.4.0-SNAPSHOT.jar
  • spark-launcher_2.10-1.4.0-SNAPSHOT.jar
  • spark-mllib_2.10-1.4.0-SNAPSHOT.jar
  • spark-network-common_2.10-1.4.0-SNAPSHOT.jar
  • spark-network-shuffle_2.10-1.4.0-SNAPSHOT.jar
  • spark-repl_2.10-1.4.0-SNAPSHOT.jar
  • spark-sql_2.10-1.4.0-SNAPSHOT.jar
  • spark-streaming_2.10-1.4.0-SNAPSHOT.jar
  • spire-macros_2.10-0.7.4.jar
  • spire_2.10-0.7.4.jar
  • stax-api-1.0.1.jar
  • stream-2.7.0.jar
  • tachyon-0.5.0.jar
  • tachyon-client-0.5.0.jar
  • uncommons-maths-1.2.2a.jar
  • unused-1.0.0.jar
  • xmlenc-0.52.jar
  • xz-1.0.jar
  • zookeeper-3.4.5.jar

[SparkQA]

Test build #30322 has finished for PR 5358 at commit 6014acc.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[SparkQA]

Test build #30328 has finished for PR 5358 at commit 54abc3a.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[dreamquster]

What's wrong with the code in PruningSuite?

createPruningTest("Column pruning - explode with aggregate",
    "SELECT name, sum(d) AS sumd FROM person LATERAL VIEW explode(data) d AS d GROUP BY name",
    Seq("name", "sumd"),
    Seq("name","data"),
    Seq.empty)

  createPruningTest("Column pruning - outer explode with limit",
    "SELECT name FROM person LATERAL VIEW OUTER explode(data) outd AS d" +
      " where  name < \"C\" limit 3",
    Seq("name"),
    Seq("name", "data"),
    Seq.empty)

  createPruningTest(s"Column pruning - select all without explode optimze - query test",
    "SELECT * FROM person LATERAL VIEW OUTER explode(data) outd AS d WHERE 20 < age",
    Seq("name", "age", "data", "d"),
    Seq("name", "age", "data"),
    Seq.empty)

jenkins report -> Error: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

[marmbrus]

Just look for Project on top of generate instead of attempting to walk the query tree. You can assume other rules will push projects down to you.

[dreamquster]

Generate block the process of collectProjectsAndFilters in PhysicalOperation.
Except that Project on top of generate, the expression named Filter may also be on the top of generate. How to solve this scene?
Project
Filter
Generate(explode)
Are there many Filters between Project and Generate?

[marmbrus]

It is possible that we need several rules to accomplish all of the various optimizations. However, as its written now this is trying to do too much and as a result is too hard to follow.

[AmplabJenkins]

Can one of the admins verify this patch?

[rxin]

@cloud-fan can you take this over?

[andrewor14]

Looks like #8286 already supersedes this. I think we can safely close this patch.

[rxin]

I'm going to close this pull request.