Skip to content

[SPARK-8405][Doc] Add how to view logs on Web UI when yarn log aggregation is enabled #7463

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
carsonwang wants to merge 4 commits into from
Closed

[SPARK-8405][Doc] Add how to view logs on Web UI when yarn log aggregation is enabled #7463

Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e5775c1
Update doc about how to view the logs on Web UI when yarn log aggrega…
carsonwang Jul 17, 2015
5a95046
Update the text in the doc
carsonwang Jul 20, 2015
74df3a1
address comments
carsonwang Jul 22, 2015
274c054
Minor text fix
carsonwang Jul 25, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/running-on-yarn.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,9 +68,9 @@ In YARN terminology, executors and application masters run inside "containers".

yarn logs -applicationId <app ID>

will print out the contents of all log files from all containers from the given application. You can also view the container log files directly in HDFS using the HDFS shell or API. The directory where they are located can be found by looking at your YARN configs (`yarn.nodemanager.remote-app-log-dir` and `yarn.nodemanager.remote-app-log-dir-suffix`).
will print out the contents of all log files from all containers from the given application. You can also view the container log files directly in HDFS using the HDFS shell or API. The directory where they are located can be found by looking at your YARN configs (`yarn.nodemanager.remote-app-log-dir` and `yarn.nodemanager.remote-app-log-dir-suffix`). The logs are also available on Web UI. You need have both the Spark history server and the MapReduce history server running and configure `yarn.log.server.url` in `yarn-site.xml` properly. The log URL on the Spark history server UI will redirect you to the MapReduce history server to show the aggregated logs.
Copy link
Contributor

@tgravescs tgravescs Jul 20, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"The logs are also available on Web UI" is pretty generic. Can we make this more specific like on the Spark Web UI under the Executors Tab


When log aggregation isn't turned on, logs are retained locally on each machine under `YARN_APP_LOGS_DIR`, which is usually configured to `/tmp/logs` or `$HADOOP_HOME/logs/userlogs` depending on the Hadoop version and installation. Viewing logs for a container requires going to the host that contains them and looking in this directory. Subdirectories organize log files by application ID and container ID.
When log aggregation isn't turned on, logs are retained locally on each machine under `YARN_APP_LOGS_DIR`, which is usually configured to `/tmp/logs` or `$HADOOP_HOME/logs/userlogs` depending on the Hadoop version and installation. Viewing logs for a container requires going to the host that contains them and looking in this directory. Subdirectories organize log files by application ID and container ID. The logs are also available on Web UI even without running the MapReduce history server.
Copy link
Contributor

@tgravescs tgravescs Jul 20, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

again can we be a bit more specific here and make it Spark Web UI or again spark web ui under the executors tab. Perhaps also rephrase a bit:
and doesn't require running the MapReduce history server.


To review per-container launch environment, increase `yarn.nodemanager.delete.debug-delay-sec` to a
large value (e.g. 36000), and then access the application cache through `yarn.nodemanager.local-dirs`
Expand Down