Utilities for IPython on Sense

This package complements Sense's REST API by wrapping and simplifying some of the most common operations, such as launching and stopping worker dashboards. Its primary purpose is to support other packages that implement higher-level approaches to cluster computing.

Installation

This package is preinstalled on Sense. You can import it with

import sense

To install it elsewhere use

pip install sense

Example

This example launches several worker dashboards and communicates with them using ZeroMQ over the project's private network.

import time
import sense

n_workers = 3

# Use 'install' to install package 'pyzmq' from PyPI to the project.
sense.install('pyzmq')
import zmq

# Create the ZeroMQ server.
context = zmq.Context()
socket = context.socket(zmq.REP)

# Use 'get_network_info' to find out the private IP address of the current
# dashboard in the project's virtual private network.
address = "tcp://" + sense.network_info()['project_ip'] + ":5000"
socket.bind(address)

# Define code the worker dashboards should execute on startup.
# Each worker will attempt to connect to the ZeroMQ server whose 
# address is stored in its 'SERVER' environment variable and then will send
# a message to the server.
worker_code = """
import os
# Because pyzmq was previously installed to the project, workers don't
# need to reinstall it.
import zmq
import sense

# Connect to the master
context = zmq.Context()
print "Connecting to master..."
socket = context.socket(zmq.REQ)
socket.connect(os.environ['SERVER'])

# Send a message
socket.send("Sense is so easy!")

# Wait for a reply
message = socket.recv()
print "Received reply: ", message
"""

# Use 'launch_workers' to start three small worker dashboards. The above
# code is sent to each, and the current dashboard's project IP address is
# stored in each worker's environment as 'SERVER', so each worker will contact
# the current dashboard.
workers = sense.launch_workers(n=n_workers, 
    size=0, 
    code=worker_code, 
    env={"SERVER": address})

# Listen for worker messages.
for i in range(0, n_workers):
    #  Wait for next request from client
    message = socket.recv()
    print "Received request: ", message
    time.sleep (1)  
    socket.send("I agree.")

sense.stop_workers()

API

install

Installs a Python package to the project with pip using the user scheme.

import sense
sense.install(package_name, flags=[], arguments={})

If you prefer, you can also install packages by running a shell command from IPython using the ! prefix:

!pip install pyzmq --user

Options:

• flags: A list of strings to pass to pip as flags. For example, ["U", "use-mirrors"] would translate to the command-line flags -U --use-mirrors.
• arguments: A dict containing arguments to pass to pip. For example, {"d": "./downloads", "mirrors": "http://URL"} would translate to the command-line arguments -d ./downloads --mirrors=http://URL.

Once installed, any of the project's dashboards can import the package.

network_info

Returns the current dashboard's contact information in a dict with keys public_dns, public_port_mapping, ssh_password and project_ip.

import sense
network_info = sense.network_info()

Every project has its own virtual private network (VPN). The project IP address is bound to the project VPN and is only accessible to other dashboards in the same project. The project VPN makes it easy to use cluster computing frameworks that lack built-in security features, such as MPI.

The public DNS hostname, public port mapping and SSH password describe how the current dashboard can be contacted from outside the project. The public port mapping is a dict whose keys and values are integers. Only ports that are keys of the public port mapping can be accessed via the public DNS hostname. If you run a service on port 3000, for example, it can be accessed from anywhere on the internet on the public DNS hostname and port public_port_mapping[3000].

If required, you can SSH to dashboards using the public DNS hostname and port public_port_mapping[22] with username "sense" and the SSH password.

launch_workers

Launches worker dashboards into the cluster.

import sense
 worker_info = sense.launch_workers(n, 
    size=0, 
    engine="sense-ipython-engine", 
    script="", 
    code="", 
    env={})

In Sense, a cluster is a group of dashboards with the same master dashboard. Worker dashboards multiplex their outputs to the master and are cleaned up automatically when the master is stopped or fails. These features make it easy to manage, monitor and debug distributed applications on Sense.

The parameters are:

• n: The number of workers to launch.
• size (optional): The size of the workers, 0 to 16.
• engine (optional): The name of the npm module to use as the engine. Defaults to "sense-ipython-engine", but workers can run other engines too.
• script (optional): A script file that the worker should execute on launch. The path is relative to the project's home folder.
• code (optional): Code that the worker should execute on launch. If both are provided, script has precedence over code.
• env (optional): A dict containing environment variables that should be set on the workers before any code is executed. This is the preferred way to send a master's contact information information to workers.

The return value is a list of dicts. Each dict describes one of the workers that was launched and contains keys such as "id", "engine", "status", etc. The full format is documented here.

list_workers

Returns information on the worker dashboards in the cluster in a list of dicts like those returned by launch_workers.

import sense
worker_info = sense.list_workers()

get_master

Returns information on the cluster's master dashboard in a dict like the ones returned by launch_workers.

import sense
master_info = sense.get_master()

stop_workers

Stops worker dashboards.

Dashboards' numerical IDs are available at key id in the dicts returned by list_workers and launch_workers. The return value is a dict of the same type.

import sense

# To stop specific workers:
worker_info = sense.stop_workers(id1, id2, ...)

# To stop all workers in the cluster:
worker_info = sense.stop_workers()

get_auth

Returns authentication information for the REST API as a dict with keys "user" and "password".

Sense's REST API gives you complete programmatic control over virtually every aspect of Sense. Most REST calls require Basic Authentication. To make authenticated REST calls, supply the information returned by get_auth your HTTP client of choice, such as the Python requests package.

By default get_auth uses the environment variable SENSE_API_TOKEN for authentication. This token restricts access to the current project. For access across projects, you can pass in credentials manually or set SENSE_USERNAME and SENSE_PASSWORD in the environment. To better understand these options, read the Understanding Project Security documentation.

Authenticated REST Example

This example retrieves information about the current project:

import sense
import requests
auth = sense.get_auth()
url = "https://api.senseplatform.com/users/" + os.environ["SENSE_OWNER_ID"] + \
   "/projects/" + os.environ["SENSE_PROJECT_ID"]
response = requests.get(url, auth=(auth["user"], auth["password"])).json()

The environment variables used in this example are common to all dashboards, across all engines, and are documented here.

Rich Dashboard Output

IPython provides its own rich display system, so unlike its R and JavaScript counterparts this package does not provide any rich output functions. Use IPython's rich display system to display HTML, images and more in a dashboard.

Support

• Documentation:

• Email:

• Twitter:

• Google Group:

• IRC: #senseplatform on irc.freenode.net

License

MIT