Skip to content

[Doc] [SQL] Addes Hive metastore Parquet table conversion section #5348

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
liancheng wants to merge 4 commits into from
Closed

[Doc] [SQL] Addes Hive metastore Parquet table conversion section #5348

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
50675db
Addes Hive metastore Parquet table conversion section
liancheng Apr 3, 2015
756e660
Adds Python snippet for metadata refreshing
liancheng Apr 3, 2015
4c9847d
Resorts to SQL for Python metadata refreshing snippet
liancheng Apr 3, 2015
42ae0d0
Adds Python `refreshTable` snippet
liancheng Jun 23, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Adds Python refreshTable snippet
  • Loading branch information
@liancheng
liancheng committed Jun 23, 2015
commit 42ae0d0d1e463b57ba37ca7b826ea01b9288e27b
15 changes: 12 additions & 3 deletions docs/sql-programming-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1271,6 +1271,15 @@ sqlContext.refreshTable("my_table")

</div>

<div data-lang="python" markdown="1">

{% highlight python %}
# sqlContext is an existing HiveContext
sqlContext.refreshTable("my_table")
{% endhighlight %}

</div>

<div data-lang="sql" markdown="1">

{% highlight sql %}
Expand Down Expand Up @@ -1518,8 +1527,8 @@ This command builds a new assembly jar that includes Hive. Note that this Hive a
on all of the worker nodes, as they will need access to the Hive serialization and deserialization libraries
(SerDes) in order to access data stored in Hive.

Configuration of Hive is done by placing your `hive-site.xml` file in `conf/`. Please note when running
the query on a YARN cluster (`yarn-cluster` mode), the `datanucleus` jars under the `lib_managed/jars` directory
Configuration of Hive is done by placing your `hive-site.xml` file in `conf/`. Please note when running
the query on a YARN cluster (`yarn-cluster` mode), the `datanucleus` jars under the `lib_managed/jars` directory
and `hive-site.xml` under `conf/` directory need to be available on the driver and all executors launched by the
YARN cluster. The convenient way to do this is adding them through the `--jars` option and `--file` option of the
`spark-submit` command.
Expand Down Expand Up @@ -1962,7 +1971,7 @@ options.
#### DataFrame data reader/writer interface

Based on user feedback, we created a new, more fluid API for reading data in (`SQLContext.read`)
and writing data out (`DataFrame.write`),
and writing data out (`DataFrame.write`),
and deprecated the old APIs (e.g. `SQLContext.parquetFile`, `SQLContext.jsonFile`).

See the API docs for `SQLContext.read` (
Expand Down