Before you can create a cluster you’ll need to sign up for an account. You can do that here.
You’ll receive a confirmation email within a few minutes, and can activate your account by clicking the included link.
If you already have an account, you can sign in at batch-user.crumpington.com.
After you sign in, you’ll see the following page:
You can click the
Create Cluster button to create a new cluster.
Note: If your account has a balance of zero or less, you’ll need to click
Account button in the toolbar and make a deposit before you can create a
After clicking the
Create Cluster button, you’ll be presented with a number
of options for your new cluster. You can read about these options in the
Most of these options can be modified later, but you should pay attention to the two that cannot be changed:
Region: This is the physical location of your cluster.
Shared Volume Size: This is the size of the file system that is shared between your compute nodes.
When setting up a new cluster, you’ll probably want to use the smallest head and compute node sizes. Once your cluster is configured to your liking, you can select larger head and compute node sizes.
You’ll also want to set both
Minimum Number of Nodes and
Maximum Number of
Nodes to 1, so your cluster will have one persistent compute node. Once your
cluster is configured, you can set
Minimum Number of Nodes to zero if you’d
rather not have a node on standby when the job queues is empty.
You’ll want to be sure to copy your ssh public key into the
Keys field. On Linux, you can find your public key in your home directory
Without a valid SSH key, you’ll be unable to sign in to a compute node to configure the cluster.
Create button to create the cluster.
After your cluster is created you’ll be taken the a page displaying the cluster’s details:
You’ll need to click the
Start button to start the cluster’s head node. It
should take a minute or two for the head node and a single compute node to
start up. If you’re using a large shared volume, it make take a bit longer to
start the cluster the first time because the shared volume needs to be
Once your cluster is active, and you set the
Minimum Nodes setting greater
than zero, you should see a compute node appear at the bottom of the cluster’s page:
It might take a few minutes for the node to become available, so be patient.
If you’ve configured your SSH authorized keys properly, you’ll now be able to log in to the compute node:
Your compute nodes will always boot to the same state because they don’t have
any persistent storage. All of your persistent data should be stored on the
shared file system, which is mounted in
When a compute node boots, it will run scripts located in the
/mnt/shared/init/ directory using the
run-parts program. These scripts will be
run with root privileges. For example, if you need ImageMagick installed on
your compute nodes, you could create a script:
#!/bin/bash apt-get update -y apt-get install -y imagemagick
and save it into
Make sure the script is executable:
chmod +x /mnt/shared/init/100-install-imagemagick
Now every time a compute nodes is started, it will automatically install ImageMagick.
You’re free to put any data you’d like on the shared filesystem, but keep in
mind that the directories
/mnt/shared/init/ are used
by the system.
Once you have your init scripts written and made executable, you can test that the startup process works properly by setting your cluster’s node timeout to 60 seconds, and shutting down your compute node:
sudo shutdown -h now
In a few minutes, your cluster will spawn a new compute node to replace the “failed” node that you shut down. If you sign in to your new compute node (find the address as you did for the original compute node) you should find that your init scripts were run.
Note: It’s possible to sign in to a compute node before the initialization process is finished. If your init script wasn’t run, wait a few minutes and check your system’s log files for details.
In order to interact with your cluster, you’ll need to download the
command line tool to your PC. (In the future we plan to provide libraries for
various languages.) You can download
bclient for your architecture
For the remainder of this article, we’ll assume that
bclient is in your path,
Any interaction with your cluster will require your cluster’s API key. This is shown in the cluster’s details in the web interface. Anyone with a cluster’s API key can submit, cancel, or delete jobs, so keep your API key safe.
Below we’ll assume that your API key is stored in an environment variable:
Before doing anything with your cluster, you’ll need to initialize your client:
bclient init -key $KEY
The client will contact our master server and fetch the IP address for the cluster’s head node and the certificate necessary to encrypting communication.
Create a new job with the
bclient job-create -key $KEY
This command will print the new job’s ID to the standard output.
Create a local directory to store your new job. We’ll name it
Create job’s run script in
jobdir/run.sh with your job:
#!/bin/bash convert -density 300 input.pdf page-%02d.jpg
For this example we’ll convert a PDF into images. We save a pdf file as
Upload the job and associated files:
bclient job-upload -key $KEY -job-id $JOBID -job-dir ./jobdir
You can replace
$JOBID with your job’s unique ID.
To place the job in the queue, run:
bclient job-enqueue -key $KEY -job-id $JOBID
You can check a job’s status using the
bclient job-info -key $KEY -job-id $JOBID
You can also list queued and running jobs using the
bclient queue-list-queued -key $KEY
This and other listing commands return the comma-separated fields: job ID, tag, priority, and the job’s state.
Once the job’s state becomes
completed, you can download the resulting files
bclient job-download -key $KEY -job-id $JOBID -job-dir ./jobdir
Once you’ve retrieved the job’s results, you should delete the job files on the cluster to avoid filling up the shared file system:
bclient job-delete -key $KEY -job-id $JOBID