NAV Navbar
python javascript
  • Introduction
  • Getting Started
  • Concepts
  • Running Jobs
  • Configuring Jobs
  • Storing Data
  • Running a Web Crawler
  • Security
  • Client Reference
  • Errors
  • Introduction

    Code examples are shown in this column.

    Welcome! AegisBlade lets you deploy & run your code with two extra lines of code.

    Sign up to get an API key or contact us if you have any questions.

    But How?

    Installed as a dependency of your project, AegisBlade introspects the application to determine its necessary runtime components, including dependencies.

    This allows us to deploy, run, & scale any part of your codebase with minimal configuration.

    Never Write YAML

    Any configuration that is required for your application is placed directly in your code.

    It is "Infrastructure as Code" taken to the limit.

    Runs Where You Do

    It links to your cloud account (AWS for example), provisions servers as you require, deploys & runs your code, saves logs & return values, and cleans up when the code is completed.

    The following cloud providers are currently supported:

    Supported Languages

    Language Versions
    Python 2.7 3.4 - 3.7
    Node.js 8 - 12

    Support for other languages is under development.

    Send us an email if you'd like to see your favorite language supported.

    Getting Started

    Use the tabs above to select your programming language.

    Getting Started with AegisBlade only takes a few minutes to setup. Afterwards, running code on a server will take a few seconds.

    View code examples in the dark area to the right, and switch the programming language of the examples with the tabs in the top right.

    See additional examples on Github.

    Sign Up

    Register to get an API key.

    Sign up for AegisBlade to get your API key.

    When prompted, follow the directions to link a cloud provider account (ex. AWS). Upon receiving the confirmation email, click the link to confirm your email address and then the API will be accessible.

    Installation

    To install the AegisBlade library, open a shell, go to your project directory and type:

    pip install aegisblade
    
    npm install aegisblade
    

    Use your language's package manager to install the AegisBlade client library.

    The library should be installed as a part of your project's dependencies, not globally.

    Authentication

    Set the AEGISBLADE_API_KEY environment variable or set your api key in code.

    $ export AEGISBLADE_API_KEY="my-api-key"
    

    You should have already signed up for your AegisBlade API key.

    To use the api key, you may set it in code or set the environment variable AEGISBLADE_API_KEY.

    Setting the environment variable is the recommended method.

    Run "Hello World"

    Run this code to create your first job:

    from aegisblade import aegisblade
    
    def hello_world():
        return "Hello World"
    
    if __name__ == "__main__":
        aegisblade.set_api_key("my-api-key")
    
        job = aegisblade.run(lambda: hello_world())
    
        print("Job Id: {0}".format(job.id))
        print("Waiting for hte job to finish running...")
    
        return_value = job.get_return_value()
    
        print("Job Return Value: {0}".format(return_value)
    
    const {aegisblade} = require('aegisblade');
    
    function helloWorld() {
        return "Hello World"
    }
    
    // Ensure target functions are exported in Node.
    module.exports = {helloWorld};
    
    if (require.main === module) {
        (async () => {
            aegisblade.setApiKey("my-api-key");
    
            let job = await aegisblade.run(helloWorld);
    
            console.log(`Job Id: ${job.id}`);
            console.log("Waiting for the job to finish running...");
    
            let jobReturnValue = await job.getReturnValue();
    
            console.log(`Job Return Value: ${jobReturnValue}`);
        })();
    }
    

    For the first job, we'll keep it simple and run a function that simply returns "Hello World".

    When you call the AegisBlade run() function, we upload your application files, provision an isolated environment, deploy your code, and then run it. The return value of run() is an object that you can use to save the job id for later use, fetch the return value, and get logs.

    In this case we are running a job that only returns a string "Hello World", waiting until it returns, and then printing the return value.

    Concepts

    Applications

    You can view files to be uploaded prior to calling run()

    upload_files = aegisblade.get_upload_files()
    
    for upload_file in upload_files:
        print(upload_file)
    
    let uploadFiles = aegisblade.getUploadFiles();
    
    for (var uploadFile of uploadFiles) {
        console.log(uploadFile);
    }
    

    An application is a collection of files and package-manager dependencies that will be used to create a deployable image of your code.

    The first time you run a job, the AegisBlade client will upload the necessary files and information required to create a deployable image.

    These files are cached, so changes to the application will only require uploading the changed files.

    Application Image Builds

    After the necessary files are uploaded, AegisBlade will provision a host on your cloud provider to build the application image.

    The image is saved and can be deployed as many times as is necessary for your use.

    While AegisBlade builds the application image, jobs can be queued to run on that application after it is ready.

    The length of time required to build the application image depends primarily upon the aggregate size of the uploaded files and the amount of time necessary to install the dependencies of your application. A typical application will build in about a minute.

    Jobs

    Jobs are conceptually a task to run some part of an application.

    An application establishes a deployable image of your code, and a job points to a function in that code to go run.

    After an application is built, jobs will run as soon as a host can be provisioned and the application image deployed. The time of this depends upon your cloud provider and the size of your application image, but typically is under a minute.

    In situations where a host is already provisioned (and has resources available), and the application image has already been deployed to that host, a job will run within seconds of being created.

    Capabilities

    Capabilities are used to request functionality that cannot be defined through the native package manager (pip in python's case).

    They may change aspects of the base image, the way a job is executed, or even components of the underlying Host OS.

    Name Description
    xvfb Requests that xvfb be used to make an X Display available. Jobs are run using xvfb-run.
    chrome Requests that the application have the google-chrome and chromedriver binaries available on the PATH. Also implies Xvfb.
    firefox Requests that the application have the firefox and geckodriver binaries available on the PATH. Also implies Xvfb

    Data Store

    For any data storage needs outside of the job's return value, AegisBlade provides the DataStore API which allows for storage of arbitrary data and is accessible to any job or your local machine.

    See the data storage section for more detail.

    Hosts

    Hosts are provisioned on an as-needed basis, and are cleaned up when they are no longer running any jobs.

    Host Drivers

    Jobs may be configured to use different host drivers that utilize different cloud platforms to provision resources.

    Different options may be provided for the host driver such as instance type, or region.

    Currently, ec2 is the only host driver available.

    Host Affinity

    Assigning "host1" to jobs 1-4 and "host2" to jobs 5-8 will force both sets of jobs to run on different hosts regardless of other factors.

    Host affinity is a string label defined in a job's configuration used to group jobs on a host(s).

    If specified, the job will only run on hosts with the same host affinity label.

    The labels are applied at the time a host is provisioned, and are not limited to a single host. When a job is run, the host scheduler will determine if there are any hosts with a matching affinity label and available resources. If there are none, a new host with the specified host affinity label will be provisioned.

    Running Jobs

    Jobs are created by calling aegisblade.run()

    Jobs run a target function in any part of your application with a set of arguments.

    After a job is created, it will be visible on the web dashboard. From there you can view its status, error details, and output logs.

    Job Status

    Fetch a job's status using a job's id:

    job_id = "some-jobs-id"
    job = aegisblade.job(job_id)
    
    print(job.get_status().__dict__)
    
    let jobId = "some-jobs-id";
    let job = await aegisblade.job(jobId);
    
    console.dir(await job.getStatus());
    

    A job's status can be fetched anytime using the client.

    Job Return Values

    Fetch return values using a job's id:

    job_id = "some-jobs-id"
    job = aegisblade.job(job_id)
    
    # Calling .get_return_value() will wait for job completion before returning.
    print(job.get_return_value())
    
    let jobId = "some-jobs-id";
    let job = await aegisblade.job(jobId);
    
    // Calling .getReturnValue() will wait for job completion before returning.
    console.log(await job.getReturnValue());
    

    The return value of a job is serialized and saved in the linked cloud file store (ex. S3).

    This can be fetched any time using the client.

    If the job throws an exception while running, the return value will be set to the exception object.

    Job Logs

    Fetch logs using a job's id:

    job_id = "some-jobs-id"
    job = aegisblade.job(job_id)
    
    print(job.get_logs())
    
    let jobId = "some-jobs-id";
    let job = await aegisblade.job(jobId);
    
    console.log(await job.getLogs());
    

    A job's logs are saved in the linked cloud file store (ex. S3).

    They can be fetched any time using the client, or viewed on the web dashboard.

    Job Function Arguments

    Function arguments are serialized & delivered to the job when it runs.

    from aegisblade import aegisblade
    
    def sendEmailFunction(emailAddress):
        print(emailAddress)
    
    if __name__ == "__main__":
        aegisblade.set_api_key("my-api-key")
    
        job = aegisblade.run(lambda: sendEmailFunction("[email protected]"))
    
        print("Job Id: {0}".format(job.id))
        print("Waiting for hte job to finish running...")
    
        return_value = job.get_return_value()
    
        print("Job Return Value: {0}".format(return_value)
    
    const {aegisblade} = require('aegisblade');
    
    function sendEmailFunction(emailAddress) {
        print(emailAddress);
    }
    
    // Ensure target functions are exported in Node.
    module.exports = {sendEmailFunction};
    
    if (require.main === module) {
        (async () => {
            aegisblade.setApiKey("my-api-key");
    
            let job = await aegisblade.run(sendEmailFunction, ["[email protected]"]);
    
            console.log(`Job Id: ${job.id}`);
            console.log("Waiting for the job to finish running...");
    
            let jobReturnValue = await job.getReturnValue();
    
            console.log(`Job Return Value: ${jobReturnValue}`);
        })();
    }
    

    When arguments are provided to a job's target function, they are serialized as a part of the job.

    It's recommended to keep function arguments to simple types (like integer & string), and to keep the overall size small.

    If more complex objects are passed as arguments, the client will make a best attempt to serialize it, and throw an exception if there is any error.

    Use the Data Store API if you need to send larger amounts (>= 1MB) of data to your jobs.

    Nested Jobs

    Every job has the AEGISBLADE_API_KEY environment variable automatically set.

    from aegisblade import aegisblade
    
    def job2():
        print("I was started by job1!")
    
    def job1():
        aegisblade.run(lambda: job2())
    
    if __name__ == "__main__":
        aegisblade.run(lambda: job1())
    
    const {aegisblade} = require("aegisblade");
    
    function job2() {
        console.log("I was started by job1!");
    }
    
    function job1() {
        await aegisblade.run(job2);
    }
    
    module.exports = {job1, job2};
    
    if (require.main === module) {
        (async () => {
            await aegisblade.run(job1);
        })();
    }
    

    Jobs may spawn any amount of other jobs.

    Every job is issued a special API Key, and the AEGISBLADE_API_KEY environment variable is automatically set.

    Runaway Infinite Job Loops

    It may happen that a job is accidently made in a way that it spawns jobs that spawn jobs in an infinite loop.

    We have internal mechanisms to stop this infinite loop by tracking the root job. We are working on exposing these internal mechanisms via the API.

    There are currently two options to stop the infinite loop:

    Configuring Jobs

    from aegisblade import JobConfig, Capability
    
    # An example showing ever job config method.
    job_config = JobConfig() \
        .with_memory(128) \
        .with_host_driver("ec2", {
            'region': 'us-east-1',      # Optional, default is set in the account
            'instanceType': 'm4.large', # Optional, default is set in the account
            'useSpotInstance': True,    # Optional, default is True
            'maxSpotPrice': '0.07'      # Optional, default is the on-demand price
        }) \
        .with_host_affinity("aegisblade-test-chrome-hosts") \
        .with_capability(Capability.chrome) \
        .with_library("my-library-package", "../../my-library") \
        .with_extra_file("./datadir")
    
    aegisblade.run(lambda: targetFunc(), job_config())
    
    
    const {aegisblade, JobConfig, Capability} = require('aegisblade');
    
    let jobConfig = new JobConfig()
        .withMemory(128)
        .withHostDriver("ec2", {
            region: 'us-east-1',       // Optional, default is set in account
            instanceType: 'm4.large',  // Optional, default is set in account
            useSpotInstance: true,     // Optional, default is true
            maxSpotPrice: '0.07'       // Optional, default is on-demand price
        })
        .withHostAffinity("aegisblade-test-chrome-hosts")
        .withCapability(Capability.chrome)
        .withLibrary("../../my-library")
        .withExtraFile("./datadir");
    
    await aegisblade.run(targetFunc, [], jobConfig);
    
    

    When calling aegisblade.run() an optional job configuration object may be passed to configure various aspects of how the job will be run.

    Constructing the job configuration is different in each language, but each is based around a similar object model.

    Job Config Property Description
    memory Specifies the amount of memory in MB required by the job. Used to determine how many jobs will be run per host. Specifying an amount greater than the memory available in the specified instance type in will not change the instance type to accomodate the greater amount of memory, but will force only one job to run per host.
    extra_files A list of extra files or directories (inside the current working directory tree) to include as part of the upload. Usually used to include non-js content files or directories.
    libraries A list of directories (outside the current working directory tree) to make available for import by the application at runtime. Usually used for project-to-project dependencies or npm packages installed from a local source.
    capabilities An array of capabilities required by this job.
    host_affinity A string label used to group jobs on a host(s).
    host_driver The backing driver to use in launching the hosts for the job. ("ec2" only currently).
    host_driver_options A set of options configuring the specified host driver.
    host_driver_options.instance_type The instance type to use when launching the job. For example, if you are using the "ec2" host driver you may specify "m4.large" for the instance type.
    host_driver_options.region The region in which to launch the job's host and other resources. (ex. "us-east-1" if using "ec2" host driver)
    host_driver_options.use_spot_instance (default=true) Whether or not to use spot instances (or the equivalent on a given host driver) when provisioning the jobs.
    host_driver_options.max_spot_price (default=on demand price) If using spot instances, this option defines the maximum price to pay per hour for a given spot instance.

    Host Drivers

    A host driver manages all aspects of creating, maintaining, and cleaning up provisioned hosts.

    The following host drivers are available:

    Host Driver Description
    ec2 (default) Uses AWS EC2 to provision hosts. Provisions spot instances by default.

    More host drivers are under development. Email us if you'd like to see your favorite cloud platform added.

    Libraries

    Example adding libraries to the job config.

    from aegisblade import JobConfig
    
    jobConfig = JobConfig() \
        .with_library("my-library-package", "../../my-library") \
        .with_library("my-other-package", "../../my-other-library")
    
    const {JobConfig} = require('aegisblade');
    
    let jobConfig = new JobConfig()
        .withLibrary("../../my-library")
        .withLibrary("../../my-other-library");
    

    Libraries are dependencies of your application that exist on your local machine and outside of the current project.

    Each language has slight nuances with respect to how libraries are handled, described below.

    Python Libraries

    In python, libraries may be simple directories full of .py files, or full-fledged packages with a setup.py at the root.

    In either case, a package name must be provided when adding the library. If the package name exists in the current environment, it will be excluded from download during the application build (so don't use overlapping names unless it's intentional).

    All libraries will be uploaded as a part of your application, and made available for import by placing the directory on sys.path.

    Relative imports of a library (from ..my-library import func) will not work in job. We recommend instead doing a local pip install ../my-library to keep consistency between how the library can be used locally and in job (from my-library import func). In cases where a local pip install is not possible, you may use the PYTHONPATH environment variable to add the library to your local sys.path.

    Nodejs Libraries

    In node.js, libraries are expected to be npm style package directories with a package.json file in the root directory.

    The package name will be automatically detected from the package.json file, and if this package is already installed in your current project, it will be excluded from npm install during application build. In this way packages defined in job config libraries will override any remote packages with the same name.

    Libraries are placed into an archive by calling npm pack and uploaded as a part of your application.

    Extra Files/Directories

    Example adding extra files and directories.

    from aegisblade import JobConfig
    
    jobConfig = JobConfig() \
        .with_extra_file("./datadir") \
        .with_extra_file("./data.json")
    
    const {JobConfig} = require('aegisblade');
    
    let jobConfig = new JobConfig()
        .withExtraFile("./datadir")
        .withExtraFile("./data.json");
    

    Extra files are simply extra directories or files that exist in your project's directory tree that are not automatically picked up by the AegisBlade client to be included as a part of the application.

    By default, the client only searches for code files (ex. .js, .py) to include as part of the upload and this can leave certain required data files missing during runtime of the job.

    If a directory is specified, the entire directory tree (including sub-directories) will be uploaded as a part of your application.

    The path supplied will be resolved relative to the current working directory of the running process.

    Capabilities

    from aegisblade import JobConfig, Capability
    
    jobConfig = JobConfig().with_capability(Capability.chrome)
    
    const {JobConfig, Capability} = require('aegisblade');
    
    let jobConfig = new JobConfig()
        .withCapability(Capability.chrome)
    

    Capabilities are defined in the job configuration.

    Capabilities change various aspects of the way your job is executed.

    Storing Data

    An example demonstrating various ways of uploading and downloading.

    from aegisblade import aegisblade
    
    data_store = aegisblade.data("my-data-store") 
    
    # Create only needs to be called once.
    data_store.create("s3", {"region": "us-east-1"})
    
    # Upload a string
    data_store.upload("some data", "/the/data/store/path/somedata.txt")
    
    # Upload a file
    data_store.upload_file("./data.json", "/any/path/data.json")
    
    # Download the uploaded string and store it in a local file.
    data_store.download_file("/the/data/store/path/somedata.txt", "./somedata.txt")
    
    # Download the uploaded file and store its contents in variable.
    data_json_file_contents = data_store.download("/any/path/data.json")
    
    const {aegisblade} = require("aegisblade");
    
    (async () => {
        let dataStore = aegisblade.data("my-data-store");
    
        // Create only needs to be called once.
        await dataStore.create("s3", {region: "us-east-1"});
    
        // Upload a string
        await dataStore.upload("some data", "/the/data/store/path/somedata.txt");
    
        // Upload a file
        await dataStore.upload_file("./data.json", "/any/path/data.json");
    
        // Download the uploaded string and store it in a local file.
        await dataStore.download_file("/the/data/store/path/somedata.txt", "./somedata.txt");
    
        // Download the uploaded file and store its contents in variable.
        let dataJsonFileContents = await data_store.download("/any/path/data.json");
    })()
    

    The Data Store API provides an easy-to-use method of storing arbitrary data available to any job.

    It uses a specified storage provider to store files securely, and makes them available for download to clients with access to your API key.

    The Data Store API can be used in various ways:

    When creating a data store, a storage driver and options for that driver must be specified.

    The options specific to each driver are shown below.

    See More Examples.

    Uniquely Named

    You can create multiple data stores with different names.

    Every data store must be uniquely named for a specific user.

    A named data store provides a logical "drive" and can be used as a unit of organization for different types of data.

    Each data store is bound to a storage driver at the time of creation.

    Storage Drivers

    Specify the storage driver and options in .create()

    data_store.create(storage_driver, storage_driver_options)
    
    await dataStore.create(storageDriver, storageDriverOptions)
    

    A storage driver is the backing storage provider for the Data Store.

    At the time of creation, a data store is bound to a storage driver.

    Currently, AWS S3 is the only available storage driver.

    AWS S3

    Available Options

    {
        "region": "us-east-1"       // required
    }
    

    The S3 storage driver is recommended for use when using the EC2 Host Driver.

    The desired AWS Region must be specified at the time of creation.

    Accessing Data In-Job

    Jobs have a pre-set AEGISBLADE_API_KEY environment variable.

    This API key will be automatically used by the clients when accessing the data store.

    Nothing additional needs to be done to access your Data Stores in-job.

    Data Security

    The Data Store explicitly creates storage objects that are non-public.

    Uploads and downloads to/from the data store are performed using secure pre-signed URLs with a specific expiration.

    When using pre-signed URLs, any uploaded or downloaded data will not pass through AegisBlade's systems, but will be directly uploaded/downloaded to/from the cloud storage provider used by the storage driver.

    Persons with access to your API Key, or access to your cloud storage provider will have access to stored data.

    Running a Web Crawler

    Creating a web crawler with selenium or puppeteer can be done very quickly using the appropriate capabilities.

    Full example web crawler applications are provided on Github.

    Security

    We take security very seriously at AegisBlade, and dedicate effort to ensuring the integrity and safety of our systems.

    Email us at [email protected] with any questions or concerns.

    Host OS

    Each host is provisioned with a hardened version of Container Linux.

    Various types of hardening are applied including hardened SSH configurations and sysctl configurations.

    The host OS image is regularly updated and every time a host is provisioned it will automatically use the latest one.

    Isolation Model

    Every job is run in its own container, with various security options applied including resource limitation and capability restrictions.

    The job is executed as a non-root user inside the container, following current best-practices.

    Client Reference

    Errors

    If you are running into trouble, send us an email, and we'll get back to you as soon as possible.