Databricks submit jar

Deploy jar file to azure databricks cluster from local.

The idea is to push the fat jar to databrick’s dbfs using databricks CLI.

Configure databricks CLI connection

Generate and copy PAT token in databricks workspace/User Settings. Then go:

databricks configure --token

Workspace url: https://adb-3772194496043630.10.azuredatabricks.net/

Push jar file to databricks workspace

First, pack to create the JAR

cd [ProjectRoot]
sbt assembly

Then push the jar to Databricks. (Before, run rm to delete jar from databricks if it exists already).

dbfs rm dbfs:/lib/jars/Arch.Pocs.Databricks.Application.jar
dbfs cp ./target/scala-2.12/Arch.Pocs.Databricks.Application.jar dbfs:/lib/jars/Arch.Pocs.Databricks.Application.jar