Skip to content

[SPARK-1768] Make event logger use a single file. #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
vanzin wants to merge 12 commits into from
Closed

[SPARK-1768] Make event logger use a single file. #1

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
f96aa78
Separate history server from history backend.
May 7, 2014
a1d4f1e
Rudimentary paging support for the history UI.
May 8, 2014
461fac8
Ensure server.stop() is called when shutting down.
May 9, 2014
d62cd43
Better handle failures when loading app info.
May 9, 2014
8e2aad0
Fix scalastyle issues.
May 9, 2014
81b69fb
Fix a comment.
May 9, 2014
0bb7b57
Review feedback.
Jun 2, 2014
bd790a0
Initialize new app array to expected size.
Jun 2, 2014
43e270f
Cosmetic change to listing header.
Jun 3, 2014
e043700
Make class package-private.
Jun 3, 2014
9b11205
Make event logger use a single file.
May 22, 2014
9dcfab9
Make history server parse old-style log directories.
May 23, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix a comment.
  • Loading branch information
Marcelo Vanzin committed May 28, 2014
commit 81b69fbe575c19129fe85b243ba5e58f28ca636c
12 changes: 3 additions & 9 deletions core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,15 +103,9 @@ class FsHistoryProvider(conf: SparkConf) extends ApplicationHistoryProvider
}

/**
* Check for any updates to event logs in the base directory. This is only effective once
* the server has been bound.
*
* If a new completed application is found, the server renders the associated SparkUI
* from the application's event logs, attaches this UI to itself, and stores metadata
* information for this application.
*
* If the logs for an existing completed application are no longer found, the server
* removes all associated information and detaches the SparkUI.
* Builds the application list based on the current contents of the log directory.
* Tries to reuse as much of the data already in memory as possible, but not reading
* applications that hasn't been updated since last time the logs were checked.
*/
def checkForLogs() = synchronized {
lastLogCheckTime = System.currentTimeMillis
Expand Down