Skip to content

[SPARK-23099][SS] Migrate foreach sink to DataSourceV2 #20951

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jose-torres wants to merge 8 commits into from
Closed

[SPARK-23099][SS] Migrate foreach sink to DataSourceV2 #20951

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
a617186
foreach sink to v2
jose-torres Mar 6, 2018
6ddac36
Merge remote-tracking branch 'apache/master' into foreach
jose-torres Mar 14, 2018
f792d32
Merge remote-tracking branch 'apache/master' into foreach
jose-torres Mar 30, 2018
117a30a
add docs
jose-torres Apr 2, 2018
ebd2580
rename file
jose-torres Apr 2, 2018
d62ffa1
move test suite
jose-torres Apr 2, 2018
a2e9eb2
fix commit
jose-torres Apr 2, 2018
f2c3408
rename test suite
jose-torres Apr 2, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
rename file
  • Loading branch information
@jose-torres
jose-torres committed Apr 2, 2018
commit ebd25807049502fff11fd6417fc6e851f168cdce
5 changes: 2 additions & 3 deletions ...sql/execution/streaming/ForeachSink.scala → ...aming/sources/ForeachWriterProvider.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,11 @@
* limitations under the License.
*/

package org.apache.spark.sql.execution.streaming
package org.apache.spark.sql.execution.streaming.sources

import org.apache.spark.sql.{Encoder, ForeachWriter, SparkSession}
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
import org.apache.spark.sql.catalyst.encoders.encoderFor
import org.apache.spark.sql.catalyst.encoders.{encoderFor, ExpressionEncoder}
import org.apache.spark.sql.sources.v2.{DataSourceOptions, StreamWriteSupport}
import org.apache.spark.sql.sources.v2.writer.{DataWriter, DataWriterFactory, SupportsWriteInternalRow, WriterCommitMessage}
import org.apache.spark.sql.sources.v2.writer.streaming.StreamWriter
Expand Down
2 changes: 1 addition & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ import org.apache.spark.sql.execution.command.DDLUtils
import org.apache.spark.sql.execution.datasources.DataSource
import org.apache.spark.sql.execution.streaming._
import org.apache.spark.sql.execution.streaming.continuous.ContinuousTrigger
import org.apache.spark.sql.execution.streaming.sources.{MemoryPlanV2, MemorySinkV2}
import org.apache.spark.sql.execution.streaming.sources.{ForeachWriterProvider, MemoryPlanV2, MemorySinkV2}
import org.apache.spark.sql.sources.v2.StreamWriteSupport

/**
Expand Down