Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
c48a70d
Prepare for session local timezone support.
ueshin Dec 6, 2016
1d21fec
Make Cast TimeZoneAwareExpression.
ueshin Dec 6, 2016
0763c8f
Fix DateTimeUtilsSuite to follow changes.
ueshin Dec 6, 2016
449d93d
Make some datetime expressions TimeZoneAwareExpression.
ueshin Dec 8, 2016
b59d902
Fix compiler error in sql/core.
ueshin Dec 8, 2016
3ddfae4
Add constructors without zoneId to TimeZoneAwareExpressions for Funct…
ueshin Dec 9, 2016
f58f00d
Add DateTimeUtils.threadLocalLocalTimeZone to partition-reltated Cast.
ueshin Dec 13, 2016
8f2040b
Fix timezone for Hive timestamp string.
ueshin Dec 13, 2016
63c103c
Use defaultTimeZone instead of threadLocalLocalTimeZone.
ueshin Dec 13, 2016
7066850
Add TimeZone to DateFormats.
ueshin Dec 13, 2016
1aaca29
Make `CurrentBatchTimestamp` `TimeZoneAwareExpression`.
ueshin Dec 14, 2016
e5bb246
Add tests for date functions with session local timezone.
ueshin Dec 14, 2016
32cc391
Remove unused import and small cleanup.
ueshin Dec 16, 2016
f434378
Fix tests.
ueshin Dec 16, 2016
16fd1e4
Rename `zoneId` to `timeZoneId`.
ueshin Dec 19, 2016
009c17b
Use lazy val to avoid to keep creating a new timezone object (or doin…
ueshin Dec 19, 2016
a2936ed
Modify ComputeCurrentTime to hold the same date.
ueshin Dec 19, 2016
c5ca73e
Add comments.
ueshin Dec 19, 2016
b860379
Fix `Cast.needTimeZone()` to handle complex types.
ueshin Dec 19, 2016
6746265
Fix `Dataset.showString()` to use session local timezone.
ueshin Dec 19, 2016
4b6900c
Merge branch 'master' into issues/SPARK-18350
ueshin Dec 20, 2016
4f9cc40
Modify to analyze `ResolveTimeZone` only once.
ueshin Dec 24, 2016
2ca2413
Use session local timezone for Hive string.
ueshin Dec 24, 2016
c232854
Merge branch 'master' into issues/SPARK-18350
ueshin Dec 26, 2016
5b6dd4f
Merge branch 'master' into issues/SPARK-18350
ueshin Jan 5, 2017
1ca5808
Use `addReferenceMinorObj` to avoid adding member variables.
ueshin Jan 10, 2017
702dd81
Use Option[String] for timeZoneId.
ueshin Jan 10, 2017
33a3425
Update a comment.
ueshin Jan 10, 2017
5cc93e3
Fix overloaded constructors.
ueshin Jan 11, 2017
5521165
Fix session local timezone for timezone sensitive tests.
ueshin Jan 11, 2017
bd8275e
Remove `timeZoneResolved` and use `timeZoneId.isEmpty` instead in `Re…
ueshin Jan 11, 2017
183945c
Merge branch 'master' into issues/SPARK-18350
ueshin Jan 14, 2017
22a3b6e
Remove unused parameter.
ueshin Jan 16, 2017
30d51fa
Merge branch 'master' into issues/SPARK-18350
ueshin Jan 16, 2017
043ab52
Use Cast directly instead of dsl.
ueshin Jan 16, 2017
3ba5830
Merge branch 'master' into issues/SPARK-18350
ueshin Jan 22, 2017
9ab31f0
Revert unnecessary changes.
ueshin Jan 22, 2017
b954947
Use `@` binding to simplify pattern match.
ueshin Jan 22, 2017
dbb2604
Inline a `lazy val`.
ueshin Jan 22, 2017
186cd3e
Add some TODO comments for follow-up prs.
ueshin Jan 22, 2017
6631a69
Add a config document.
ueshin Jan 22, 2017
3610465
Use an overload version of `checkAnswer`.
ueshin Jan 22, 2017
c12e596
Fix CastSuite and add some comments to describe the tests.
ueshin Jan 22, 2017
8a04e80
Use None instead of null.
ueshin Jan 22, 2017
efe3aff
Add some comments to describe the tests.
ueshin Jan 22, 2017
cdbb266
Make TimeAdd/TimeSub/MonthsBetween TimeZoneAwareExpression.
ueshin Jan 22, 2017
328399a
Add comments to explain tests.
ueshin Jan 23, 2017
7352612
Modify a test.
ueshin Jan 23, 2017
b99cf79
Refine tests.
ueshin Jan 25, 2017
a85377f
Remove unnecessary new lines.
ueshin Jan 26, 2017
f0c911b
Add newDateFormat to DateTimeUtils and use it.
ueshin Jan 26, 2017
6fa1d6a
Parameterize some tests.
ueshin Jan 26, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Make CurrentBatchTimestamp TimeZoneAwareExpression.
  • Loading branch information
ueshin committed Dec 16, 2016
commit 1aaca29122d9996da4a2d60a3e9aae01b89e685e
Original file line number Diff line number Diff line change
Expand Up @@ -101,11 +101,20 @@ case class CurrentTimestamp() extends LeafExpression with CodegenFallback {
*
* There is no code generation since this expression should be replaced with a literal.
*/
case class CurrentBatchTimestamp(timestampMs: Long, dataType: DataType)
extends LeafExpression with Nondeterministic with CodegenFallback {
case class CurrentBatchTimestamp(timestampMs: Long, dataType: DataType, zoneId: String = null)
extends LeafExpression with TimeZoneAwareExpression with Nondeterministic with CodegenFallback {

def this(timestampMs: Long, dataType: DataType) = this(timestampMs, dataType, null)

override def nullable: Boolean = false

override def timeZoneResolved: Boolean = dataType != DateType || super.timeZoneResolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't a timestamp also need a timezone?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is because it is the current timestamp right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, timestamp always have the same value regardless of timezone.


override lazy val resolved: Boolean =
childrenResolved && checkInputDataTypes().isSuccess && timeZoneResolved

override def withTimeZone(zoneId: String): TimeZoneAwareExpression = copy(zoneId = zoneId)

override def prettyName: String = "current_batch_timestamp"

override protected def initializeInternal(partitionIndex: Int): Unit = {}
Expand All @@ -119,7 +128,7 @@ case class CurrentBatchTimestamp(timestampMs: Long, dataType: DataType)
def toLiteral: Literal = dataType match {
case _: TimestampType =>
Literal(DateTimeUtils.fromJavaTimestamp(new Timestamp(timestampMs)), TimestampType)
case _: DateType => Literal(DateTimeUtils.millisToDays(timestampMs), DateType)
case _: DateType => Literal(DateTimeUtils.millisToDays(timestampMs, timeZone), DateType)
}
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ class IncrementalExecution(
*/
override lazy val optimizedPlan: LogicalPlan = {
sparkSession.sessionState.optimizer.execute(withCachedData) transformAllExpressions {
case ts @ CurrentBatchTimestamp(timestamp, _) =>
case ts @ CurrentBatchTimestamp(timestamp, _, _) =>
logInfo(s"Current batch timestamp = $timestamp")
ts.toLiteral
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -482,7 +482,7 @@ class StreamExecution(
ct.dataType)
case cd: CurrentDate =>
CurrentBatchTimestamp(offsetSeqMetadata.batchTimestampMs,
cd.dataType)
cd.dataType, cd.zoneId)
}

reportTimeTaken("queryPlanning") {
Expand Down