SHS-NG M2: Store FsHistoryProvider listing data in a KVStore.

The application listing is still generated from event logs, but is now stored
in a KVStore instance. By default an in-memory store is used, but a new config
allows setting a local disk path to store the data, in which case a LevelDB
store will be created.

The provider stores things internally using the public REST API types; I believe
this is better going forward since it will make it easier to get rid of the
internal history server API which is mostly redundant at this point.

I also added a finalizer to LevelDBIterator, to make sure that resources are
eventually released. This helps when code iterates but does not exhaust the
iterator, thus not triggering the auto-close code.

HistoryServerSuite was modified to not re-start the history server unnecessarily;
this makes the json validation tests run more quickly.

common/kvstore/src/main/java/org/apache/spark/kvstore/LevelDBIterator.java

@@ -172,6 +172,18 @@ public synchronized void close() throws IOException {
     }
   }
 
+  /**
+   * Because it's tricky to expose closeable iterators through many internal APIs, especially
+   * when Scala wrappers are used, this makes sure that, hopefully, the JNI resources held by
+   * the iterator will eventually be released.
+   */
+  @Override
+  protected void finalize() throws Throwable {
+    if (db.db() != null) {
+      close();
+    }
+  }
+
   private T loadNext() {
     if (count >= max) {
       return null;

core/pom.xml

@@ -67,6 +67,11 @@
       <artifactId>spark-launcher_${scala.binary.version}</artifactId>
       <version>${project.version}</version>
     </dependency>
+    <dependency>
+      <groupId>org.apache.spark</groupId>
+      <artifactId>spark-kvstore_${scala.binary.version}</artifactId>
+      <version>${project.version}</version>
+    </dependency>
     <dependency>
       <groupId>org.apache.spark</groupId>
       <artifactId>spark-network-common_${scala.binary.version}</artifactId>

core/src/main/resources/org/apache/spark/ui/static/historypage.js

@@ -188,7 +188,7 @@ $(document).ready(function() {
         }
 
         $(selector).DataTable(conf);
-        $('#hisotry-summary [data-toggle="tooltip"]').tooltip();
+        $('#history-summary [data-toggle="tooltip"]').tooltip();
       });
     });
 });

core/src/main/scala/org/apache/spark/deploy/history/ApplicationHistoryProvider.scala

@@ -68,11 +68,19 @@ private[history] abstract class HistoryUpdateProbe {
  * @param ui Spark UI
  * @param updateProbe probe to call to check on the update state of this application attempt
  */
-private[history] case class LoadedAppUI(
+private[spark] case class LoadedAppUI(
     ui: SparkUI,
     updateProbe: () => Boolean)
 
-private[history] abstract class ApplicationHistoryProvider {
+private[spark] abstract class ApplicationHistoryProvider {
+
+  /**
+   * The number of applications available for listing. Separate method in case it's cheaper
+   * to get a count than to calculate the whole listing.
+   *
+   * @return The number of available applications.
+   */
+  def getAppCount(): Int = getListing().size
 
   /**
    * Returns the count of application event logs that the provider is currently still processing.