PushToCompute Work Flow Deployment Guide

This guide is for users, developers and administrators looking to deploy custom application work flows on JARVICE. PushToCompute™, a platform capability of JARVICE, is a method for rapid cloud deployment of containerized compute intensive applications. It leverages the declarative tooling of Docker for describing and building containers for high performance applications. Once deployed to JARVICE, the JARVICE container run-time, which is optimized for high performance and accelerated applications, powers all Work Flows available in the Nimbix Cloud.

This guide is divided into two parts, Building Work Flows, and Deploying Work Flows.

Related articles:

Building PushToCompute™ Work Flows

With PushToCompute™, Work Flows are defined using a simple, declarative repository structure centered around a Dockerfile. The Docker tooling provides a useful mechanism for describing and building Linux container images.

Types of Work Flows

Most PushToCompute™ Work Flows on JARVICE can be divided up into the following categories. Each category has slightly different goals and typically requires different types of set up.

  • Interactive Desktop Work Flows: Ideal for use with the Nimbix Desktop by prefixing your application with /usr/local/bin/nimbix_desktop to launch a desktop environment inside of a Nimbix base image to leverage the interactive capabilities of JARVICE directly in the browser.
    • Pre-/Post-processing
    • Purely graphical Work Flows
    • Hybrid Work Flows
  • Single-tenant Web App: well-suited for a url.txt which connects directly to the web application on the NAE’s public IP address
    • A web application which runs in an isolated environment.
  • Batch Jobs: well-suited for computationally intensive work flows which need to constrain inputs and outputs; ideal for turn-key solutions of solvers which used to require submitting complex batch scripts to a supercomputing queue.
    • A work flow which corresponds to running a single command to process inputs and to produce outputs.
  • Service Jobs: Some types of Work Flows are long-running service jobs. If your application cannot be run in as “serverless,” on-demand work flow, applications can be run indefinitely as a service. The lifetime of the service can be controlled by API or through the Dashboard of the JARVICE portal.
    • A long running service, such as an accelerated database

PushToCompute™ Work Flow Definition

The PushToCompute™ Work Flow exposes an application or a subset of the application’s capabilities to users on JARVICE. End users typically consume work flows from the web-based portal at https://platform.jarvice.com. The AppDef.json defines the Task Builder. JARVICE also uses the contents of AppDef.json to generate automatically a JSON-based API for your application. This allows PushToCompute™ Work Flows to be launched either from the portal, or by a simple web API call. This provides great flexibility in integrating existing web application components with a serverless high-density compute back-end powered by JARVICE.

The PushToCompute™ Work Flow Definition is a git repository consisting of the following structure. The only file that is required is the Dockerfile. However, it is highly recommended that all supporting metadata be stored in the same git repository.

Dockerfile        # Used to build the base image
README.md         # Application notes for maintainer or users (if open source)
NAE/
   help.html      # Provides HTML-based help popup
   nvidia.cfg     # Configures Nvidia OpenGL
   AppDef.png/jpg # Automatically imported as app icon
   AppDef.json    # Defines Task Builder and API on portal
   url.txt        # Defines a "Connect" URL
scripts/
   start.sh       # Application entry point script
tests/ (required for certified 3rd party apps)
   tests.sh       # Tests, which can be run in the Dockerfile during build

The Application Definition (AppDef.json)

The complete Application Definition format is documented in the JARVICE Application Definition Guide. This should be stored in NAE/AppDef.json of your git repository.

Building the Application Image

For a complete example on building an application with PushToCompute™ using Docker, see the PushToCompute™ Tutorial. Application images are defined in the Dockerfile, and its corresponding metadata for the web portal’s catalog entry can be stored in /etc/NAE/ of the application’s image. Nimbix maintains a set of official base images. These images are found in the Docker Hub registry and can be found by searching:

docker search nimbix |grep ^nimbix/

Contributed base images can be found in the jarvice namespace on the Docker Hub registry at:

docker search jarvice | grep ^jarvice/

More images, and their PushToCompute™ Work Flow Definition repositories, can be found at https://github.com/nimbix. If you are interested in contributing JARVICE base images for application development, please the guide for JARVICE base contributions.

Deploying PushToCompute™ Work Flows on JARVICE

In this section, we walk through the steps required to set up the continuous integration and deployment pipeline that triggers a deployment to JARVICE from a single git commit. You can deploy PushToCompute™ Work Flows directly to Github or Bitbucket for free continuous integration and deployment onto the JARVICE platform at https://platform.jarvice.com.

Once configured, every git push to your PushToCompute™ Work Flow Definition repository will trigger your application image to be build on Docker Hub. Once building is complete, a Docker Hub webhook triggers the application to be automatically deployed on JARVICE. If you wish to build your container images locally and push them to the Docker Hub or another Docker registry, you may skip the git repository.

Instructions for hooking up these webhooks can be found in the PushToCompute™ Tutorial.

Building Parallel Applications

For complete reference on the JARVICE runtime and building parallel applications, refer to the JARVICE Quick Start Guide.

Adding Custom Applications to the Compute Catalog

Users can create and manage their own private applications using the Create button on the PushToCompute tab. Once you have a complete work flow that you would like to make accessible to other users, please contact Nimbix Support and one of our application engineers will help you complete this process.  You can also make business arrangements with Nimbix to charge license fees for your application workflows on a per hour basis.