Quick Start

Register or Sign In

Before you can create a cluster you’ll need to sign up for an account. You can do that here.

You’ll receive a confirmation email within a few minutes, and can activate your account by clicking the included link.

If you already have an account, you can sign in at batch-user.crumpington.com.

Create a Cluster

After you sign in, you’ll see the following page:

You can click the Create Cluster button to create a new cluster.

Note: If your account has a balance of zero or less, you’ll need to click the Account button in the toolbar and make a deposit before you can create a cluster.

Choose Cluster Settings

After clicking the Create Cluster button, you’ll be presented with a number of options for your new cluster. You can read about these options in the documentation.

Most of these options can be modified later, but you should pay attention to the two that cannot be changed:

  1. Region: This is the physical location of your cluster.
  2. Shared Volume Size: This is the size of the file system that is shared between your compute nodes.

When setting up a new cluster, you’ll probably want to use the smallest head and compute node sizes. Once your cluster is configured to your liking, you can select larger head and compute node sizes.

You’ll also want to set both Minimum Number of Nodes and Maximum Number of Nodes to 1, so your cluster will have one persistent compute node. Once your cluster is configured, you can set Minimum Number of Nodes to zero if you’d rather not have a node on standby when the job queues is empty.

You’ll want to be sure to copy your ssh public key into the SSH Authorized Keys field. On Linux, you can find your public key in your home directory under .ssh/id_rsa.pub.

Without a valid SSH key, you’ll be unable to sign in to a compute node to configure the cluster.

Click the Create button to create the cluster.

Start Cluster

After your cluster is created you’ll be taken the a page displaying the cluster’s details:

You’ll need to click the Start button to start the cluster’s head node. It should take a minute or two for the head node and a single compute node to start up. If you’re using a large shared volume, it make take a bit longer to start the cluster the first time because the shared volume needs to be formatted.

Logging Into a Compute Node

Once your cluster is active, and you set the Minimum Nodes setting greater than zero, you should see a compute node appear at the bottom of the cluster’s page:

It might take a few minutes for the node to become available, so be patient.

If you’ve configured your SSH authorized keys properly, you’ll now be able to log in to the compute node:

ssh user@207.154.225.211

Configuring Compute Nodes

Your compute nodes will always boot to the same state because they don’t have any persistent storage. All of your persistent data should be stored on the shared file system, which is mounted in /mnt/shared/.

When a compute node boots, it will run scripts located in the /mnt/shared/init/ directory using the run-parts program. These scripts will be run with root privileges. For example, if you need ImageMagick installed on your compute nodes, you could create a script:

#!/bin/bash

apt-get update -y
apt-get install -y imagemagick

and save it into /mnt/shared/init/100-install-imagemagick.

Make sure the script is executable:

chmod +x /mnt/shared/init/100-install-imagemagick

Now every time a compute nodes is started, it will automatically install ImageMagick.

You’re free to put any data you’d like on the shared filesystem, but keep in mind that the directories /mnt/shared/jobs/ and /mnt/shared/init/ are used by the system.

Testing

Once you have your init scripts written and made executable, you can test that the startup process works properly by setting your cluster’s node timeout to 60 seconds, and shutting down your compute node:

sudo shutdown -h now

In a few minutes, your cluster will spawn a new compute node to replace the “failed” node that you shut down. If you sign in to your new compute node (find the address as you did for the original compute node) you should find that your init scripts were run.

Note: It’s possible to sign in to a compute node before the initialization process is finished. If your init script wasn’t run, wait a few minutes and check your system’s log files for details.

Job Management

In order to interact with your cluster, you’ll need to download the bclient command line tool to your PC. (In the future we plan to provide libraries for various languages.) You can download bclient for your architecture here.

For the remainder of this article, we’ll assume that bclient is in your path, and named bclient.

Cluster API Key

Any interaction with your cluster will require your cluster’s API key. This is shown in the cluster’s details in the web interface. Anyone with a cluster’s API key can submit, cancel, or delete jobs, so keep your API key safe.

Below we’ll assume that your API key is stored in an environment variable:

KEY=cf670503-f26c-e677-6c3b-e41aa3c1804c

Initializing

Before doing anything with your cluster, you’ll need to initialize your client:

bclient init -key $KEY

The client will contact our master server and fetch the IP address for the cluster’s head node and the certificate necessary to encrypting communication.

Creating a Job

Create a new job with the job-create command:

bclient job-create -key $KEY

This command will print the new job’s ID to the standard output.

Job Files

Create a local directory to store your new job. We’ll name it jobdir:

mkdir jobdir

Create job’s run script in jobdir/run.sh with your job:

#!/bin/bash
convert -density 300 input.pdf page-%02d.jpg

For this example we’ll convert a PDF into images. We save a pdf file as jobdir/input.pdf.

Upload Files

Upload the job and associated files:

bclient job-upload -key $KEY -job-id $JOBID -job-dir ./jobdir

You can replace $JOBID with your job’s unique ID.

Enqueue the Job

To place the job in the queue, run:

bclient job-enqueue -key $KEY -job-id $JOBID

Checking Job Status

You can check a job’s status using the job-info command:

bclient job-info -key $KEY -job-id $JOBID

You can also list queued and running jobs using the queue-list-queued command:

bclient queue-list-queued -key $KEY

This and other listing commands return the comma-separated fields: job ID, tag, priority, and the job’s state.

Download Job Result

Once the job’s state becomes completed, you can download the resulting files using the job-download command:

bclient job-download -key $KEY -job-id $JOBID -job-dir ./jobdir

Delete Job Files

Once you’ve retrieved the job’s results, you should delete the job files on the cluster to avoid filling up the shared file system:

bclient job-delete -key $KEY -job-id $JOBID