databricks-pybuilder-plugin

databricks-pybuilder-plugin

The plugin is considered to be used to deploy assets to a Databricks environment.

The plugin is activated with the following command in your build.by: > use_plugin('pypi:databricks_pybuilder_plugin')

It provides a set of tasks for uploading resources, workspaces, deploying jobs, or installing locally built whl dependencies into a databricks cluster.

Deployment to Databricks

Automated deployment is implemented on the PyBuilder tasks basis.

The list of available tasks:

1. export_workspace - exporting a workspace. A workspace is considered to be a folder in Databricks holding a notebook, or a set of notebooks.

The task uploads the src/main/scripts/ content into a Databricks workspace. It overrides files of the same names and leaves other files as is.

By default, a git branch name is used as a nested folder of a workspace for uploading the content into the Databricks workspace, if it's available. The default folder is used otherwise. Use the branch parameter to import a workspace in your own folder: >pyb export_workspace -P branch={custom_directory}

The final output path would be /team_folder/application_name/{custom_directory} in this way.

Executing the command from a master branch >pyb export_workspace

would upload the workspace files into /team_folder/application_name/master.

Here is the list of related deployment settings

All of the properties could be overridden with a -P parameter.

Usage example: >pyb export_workspace [-P env={env}] [-P branch={branch}]

Environment specific properties Disabled by default.

2. export_resources - exporting resources into dbfs. Uploads resource files into dbfs if any. Existing files are to be overridden.

Here is the list of related deployment settings

All of the properties could be overridden with a -P parameter.

Usage example: >pyb export_resources [-P env={env}] [-P branch={branch}]

3. install_library - deploying an whl-archive to a Databricks cluster. Uploads an whl archive to a Volume, and re-attaches the library to a cluster by name. Re-installing a new library version triggers the cluster starting to uninstall old libraries versions and to install a new one. Repetitive installations of a library of the same version don't start the cluster and don't upload files to s3. The library is just re-attached to a cluster in this way. Installed libraries in other branches are not affected. Reinstalling a library in the same branch deleting all files under the same branch directory in AWS s3. If a Cloud Provider other than AWS is used, feel free to override the task using the Cloud specific API.

Here is the list of related deployment settings

All of the properties could be overridden with a -P parameter.

Usage example: >pyb install_library

4. deploy_to_cluster - a full deployment to a cluster. Runs the export_resources, export_workspace, install_library in a row.

Usage example: >pyb deploy_to_cluster

5. deploy_job - deploying a job to the Databricks by name. Please, make sure that the job is created on the Databricks side.

Executes export_resources and export_workspace tasks preliminarily. Updates the existing job using a job definition file. The definition file supports the jinja2 template syntax. Please, check documentation for details: https://jinja.palletsprojects.com/en/2.11.x/

Here is the list of related deployment settings

All of the properties could be overridden with a -P parameter.

Usage example: >pyb deploy_job [-P env={env}] [-P branch={branch}]

5. post_init - initializing env and branch properties for further tasks. Executed automatically as a dependent task.

To Run a notebook with a custom dependency

1. Build the whl-archive with thepyb command.

2. Deploy all the assets using the command pyb deploy_to_cluster.

3. Get to the target folder in the Databricks workspace.

4. Attach the notebook to a cluster and run the script.

All properties list