Skip to content

Conversation

@wenfang6
Copy link
Collaborator

@wenfang6 wenfang6 commented Jan 5, 2020

What changes were proposed in this pull request?

The driver hang when we use multiple processes trying to access the XSQLSessionCatalog in XSQL, We print jstack and find a DeadLock. as follows:

"Thread-491" #895 daemon prio=5 os_prio=0 tid=0x000000000219b000 nid=0x3ebf waiting for monitor entry [0x00007f17bb2ee000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.spark.sql.xsql.XSQLExternalCatalog.setWorkingDataSource(XSQLExternalCatalog.scala:591)
	- waiting to lock <0x000000066bfb9cb8> (a java.util.concurrent.atomic.AtomicReference)
	at org.apache.spark.sql.xsql.XSQLSessionCatalog.setWorkingDataSource(XSQLSessionCatalog.scala:150)
	at org.apache.spark.sql.xsql.XSQLSessionCatalog.lookupRelation(XSQLSessionCatalog.scala:443)
	- locked <0x000000066be8f0f0> (a org.apache.spark.sql.xsql.XSQLSessionCatalog)
	at org.apache.spark.sql.xsql.XSQLSessionStateBuilder$XSQLResolveRelations$.org$apache$spark$sql$xsql$XSQLSessionStateBuilder$XSQLResolveRelations$$lookupTableFromCatalog(XSQLSessionStateBuilder.scala:283)
	at org.apache.spark.sql.xsql.XSQLSessionStateBuilder$XSQLResolveRelations$.resolveRelation(XSQLSessionStateBuilder.scala:236)
	at org.apache.spark.sql.xsql.XSQLSessionStateBuilder$XSQLResolveRelations$$anonfun$apply$1.applyOrElse(XSQLSessionStateBuilder.scala:266)
	at org.apache.spark.sql.xsql.XSQLSessionStateBuilder$XSQLResolveRelations$$anonfun$apply$1.applyOrElse(XSQLSessionStateBuilder.scala:259)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1$$anonfun$apply$1.apply(AnalysisHelper.scala:90)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1$$anonfun$apply$1.apply(AnalysisHelper.scala:90)
"Thread-490" #894 daemon prio=5 os_prio=0 tid=0x0000000002199000 nid=0x3ebe waiting for monitor entry [0x00007f17bb9f4000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction(SessionCatalog.scala:1276)
	- waiting to lock <0x000000066be8f0f0> (a org.apache.spark.sql.xsql.XSQLSessionCatalog)
	at org.apache.spark.sql.hive.HiveSessionCatalog.org$apache$spark$sql$hive$HiveSessionCatalog$$super$lookupFunction(HiveSessionCatalog.scala:131)
	at org.apache.spark.sql.hive.HiveSessionCatalog$$anonfun$3.apply(HiveSessionCatalog.scala:131)
	at org.apache.spark.sql.hive.HiveSessionCatalog$$anonfun$3.apply(HiveSessionCatalog.scala:131)
	at scala.util.Try$.apply(Try.scala:192)
	at org.apache.spark.sql.hive.HiveSessionCatalog.lookupFunction0(HiveSessionCatalog.scala:131)
	at org.apache.spark.sql.hive.HiveSessionCatalog.lookupFunction(HiveSessionCatalog.scala:117)
	at org.apache.spark.sql.xsql.XSQLSessionCatalog.org$apache$spark$sql$xsql$XSQLSessionCatalog$$super$lookupFunction(XSQLSessionCatalog.scala:801)
	at org.apache.spark.sql.xsql.XSQLSessionCatalog$$anonfun$lookupFunction$1.apply(XSQLSessionCatalog.scala:801)
	at org.apache.spark.sql.xsql.XSQLSessionCatalog$$anonfun$lookupFunction$1.apply(XSQLSessionCatalog.scala:801)
	at org.apache.spark.sql.xsql.XSQLExternalCatalog.setWorkingDataSource(XSQLExternalCatalog.scala:598)
	- locked <0x000000066bfb9cb8> (a java.util.concurrent.atomic.AtomicReference)
	at org.apache.spark.sql.xsql.XSQLSessionCatalog.setWorkingDataSource(XSQLSessionCatalog.scala:150)
	at org.apache.spark.sql.xsql.XSQLSessionCatalog.lookupFunction(XSQLSessionCatalog.scala:800)
Found 1 deadlock.

This PR solve this DeadLock.

How was this patch tested?

No UT.

override def setCurrentDatabase(dbName: String): Unit = {
withWorkingDSDB(dbName) { (_, db, _, _) =>
currentDataBase = Option(db)
synchronized{
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why there need synchronized ?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have checked previous implement.

def setCurrentDatabase(dsName: String, dbName: String): Unit = {
      setWorkingDataSource(Option(dsName)) {
        super.setCurrentDatabase(dbName)
        externalCatalog.setCurrentDatabase(dbName)
      }
}

This is the only entrance. The tread safe has provided by setWorkingDataSource, so we not need this change.

@beliefer beliefer changed the title [SQL]Fix deadLock of XSQLSessionCatalog and workingDataSource [Core] Fix deadLock between XSQLSessionCatalog and workingDataSource Jan 5, 2020
@beliefer
Copy link
Collaborator

beliefer commented Jan 5, 2020

Good issue.

@beliefer beliefer added the bug Something isn't working label Jan 5, 2020
@beliefer
Copy link
Collaborator

Merged to master and branch-0.6. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants