Run Jupyter Notebook with Spark in Docker

Goal

Run a Jupyter notebook with Apache Spark support using Docker, enabling Scala code execution and Spark monitoring.

Setup

Pull and run the Spark-enabled Jupyter container:

docker run -p 8888:8888 -p 4041:4041 jupyter/all-spark-notebook

This exposes:

• Port 8888: Jupyter interface
• Port 4041: Spark WebUI

Access Jupyter

1. Copy the authentication token from the terminal output
2. Navigate to localhost:8888 in your browser
3. Authenticate using the token

Write Spark Code

Create a new notebook with the spylon-kernel kernel to write Scala code with Spark.

Monitor Jobs

Access the Spark WebUI at localhost:4041 to monitor job execution and performance.