Skip to content

[SPARK-19335] Introduce insert, update, and upsert commands to the JdbcUtils class #16685

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ilganeli wants to merge 10 commits into from
Closed

[SPARK-19335] Introduce insert, update, and upsert commands to the JdbcUtils class #16685

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
8d499fe
Introduce insert, update, and upsert commands to the JdbcUtils class
Jan 24, 2017
89cef37
Extra line
Jan 24, 2017
e1fc6f6
Fixed merge conflicts
Jan 24, 2017
a64719b
Reverted changes to df writer
Jan 24, 2017
ca494eb
Reverted changes to savemode
Jan 24, 2017
6a2cb05
Fixed issue using mapPartitions
Jan 24, 2017
69f6939
Added spark implicits import
Jan 24, 2017
7938277
Added implicit import
Jan 24, 2017
56545ed
Import order
Jan 24, 2017
c6af861
Added implicit
Jan 24, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Import order
  • Loading branch information
Ilya Ganelin committed Jan 24, 2017
commit 56545ed88f665ed57a50a8c5d114c6ae8130eab3
9 changes: 4 additions & 5 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,10 @@ package org.apache.spark.sql.execution.datasources.jdbc
import java.sql.{Connection, Driver, DriverManager, PreparedStatement, ResultSet, ResultSetMetaData, SQLException, Statement}
import java.util.Properties

import scala.collection.JavaConverters._
import scala.util.Try
import scala.util.control.NonFatal

import org.apache.spark.TaskContext
import org.apache.spark.executor.InputMetrics
import org.apache.spark.internal.Logging
Expand All @@ -34,10 +38,6 @@ import org.apache.spark.sql.types._
import org.apache.spark.unsafe.types.UTF8String
import org.apache.spark.util.NextIterator

import scala.collection.JavaConverters._
import scala.util.Try
import scala.util.control.NonFatal

/**
* Util functions for JDBC tables.
*/
Expand Down Expand Up @@ -764,7 +764,6 @@ object JdbcUtils extends Logging {
statements: DataFrame,
batchSize: Int = DEFAULT_BATCH_SIZE,
maxConnections: Int = DEFAULT_MAX_CONNECTIONS): Unit = {
import statements.sparkSession.implicits._
// To avoid overloading database coalesce to a set number of partitions if necessary
val coalesced = if (statements.rdd.getNumPartitions > maxConnections) {
statements.coalesce(maxConnections)
Expand Down