Getting Started with deploy.dog

There's 3 simple steps to get up and running with deploy.dog.

  1. Sign up online.
  2. Register your project.
  3. Add deploy.dog config file to the root of your repository.
  4. Deploy! (We don't count this as a step )

Continue reading to see how the config file is structured…


Sign Up

deploy.dog Config File

The configuration for your deployments (apart from any secrets, such as passwords) all lives in a YAML file named deploy.dog.yml or a JSON file named deploy.dog.json in the root of your project's git repository. Keep reading to see how this file is structured. We recommend YAML over JSON for a few reasons (such as comments being possible), but it is up to you which you use.

Some of these sections may look daunting at first as all possible options are discussed. If you're concerned about the complexity, then just remember that most projects only need the basic options. See the example of a complete config file at the bottom for a simple version with only the basics options set.


Secrets

Before getting into the config, we'll introduce the concept of secrets. Most values within your config file can be substituted for a secret value. The concept is very simple; your config file is available to view by anyone who can view the code in your repository, so whilst showing your team as much of the deploy process as possible (to keep them in the loop), some things should be kept private. Good examples of this include SSH access keys, any passwords that may be needed by your migration tools and notification tokens (Slack/HipChat/etc). It is up to you what you choose to use secrets for though. Although they are highly recommended for private things, they are never compulsory.

Setting up secrets is easy. Once your project is registered on our website navigate to Your Project > More Actions > Manage Secrets. You can add secrets by giving them a name and their value. We'll then provide you with a "secret ref" to use instead of the actual secret value within your config file. If you entered a secret name of db-password, for example, then your "secret ref" for your config file would be (d.d.secret:db-password).

As SSH key pairs are one of the most common secrets to have, the Manage Secrets page of your dashboard allows you to generate an SSH Key Secret automatically. It will give you the "secret ref" to include in your config along with the Public Key to add to your authorized_keys files on the server. This saves you the hassle of generating key pairs yourself, but you can of course do that too if you prefer. Remember to add the private key as your secret value and add the public key to the server (not the other way around).

We store the actual secret values securely encrypted on our servers for you. You can update a secret value at any time on our website, but can never view the originally entered value (in fact our web servers are not even capable of decrypting it if they tried).


Servers Config

The servers section is used to define the server(s) you need to deploy to. You include how to access them and the stage and group that they belong to. At this time, we can only support Unix-based servers which we can access over SSH.

servers:
  - name: Dalmatian
    host: dalmatian.example.com
    port: 22
    user: deploydog
    group: web
    stage: production
    type: ssh
    sshKey: (d.d.secret:dalmatian-ssh-key) # You could instead provide the password, see below

  - name: Dobermann
    host: dobermann.example.com
    port: 22
    user: deploydog
    group: web
    stage: production
    type: ssh
    password: (d.d.secret:dobaermann-ssh-password) # You could instead provide the private key, see above

  - name: lb1
    host: lb1.example.com
    port: 22
    user: deploydog
    group: lb
    stage: production
    type: ssh
    sshKey: (d.d.secret:lb1-ssh-key)

  - name: lb2
    host: 10.0.60.7
    port: 22
    user: deploydog
    group: lb
    stage: production
    type: ssh
    sshKey: |
      -----BEGIN RSA PRIVATE KEY-----
      MIIEpg...INCLUDING A KEY HERE DIRECTLY IS POSSIBLE, BUT NOT RECOMMENDED. USE A SECRET...hx/YY
      -----END RSA PRIVATE KEY-----
{
  "servers": [
    {
      "name": "Dalmatian",
      "host": "dalmatian.example.com",
      "port": 22,
      "user": "deploydog",
      "group": "web",
      "stage": "production",
      "type": "ssh",
      "sshKey": "(d.d.secret:dalmatian-ssh-key)"
    },
    {
      "name": "Dobermann",
      "host": "dobermann.example.com",
      "port": 22,
      "user": "deploydog",
      "group": "web",
      "stage": "production",
      "type": "ssh",
      "password": "(d.d.secret:dobaermann-ssh-password)"
    },
    {
      "name": "lb1",
      "host": "lb1.example.com",
      "port": 22,
      "user": "deploydog",
      "group": "lb",
      "stage": "production",
      "type": "ssh",
      "sshKey": "(d.d.secret:lb1-ssh-key)"
    },
    {
      "name": "lb2",
      "host": "10.0.60.7",
      "port": 22,
      "user": "deploydog",
      "group": "lb",
      "stage": "production",
      "type": "ssh",
      "sshKey": "-----BEGIN RSA PRIVATE KEY-----\nMIIEpg...INCLUDING A KEY HERE DIRECTLY IS POSSIBLE, BUT NOT RECOMMENDED. USE A SECRET...hx/YY\n-----END RSA PRIVATE KEY-----\n"
    }
  ]
}
Parameter Type/Value Notes Required
name string Your reference for this server. It will be show in logs, error messages, etc.  Required
host string The address of the server to connect to. Can be an IP address or a fully qualified domain name (FQDN).  Required
port integer This is the port for the SSH connection. Typically this is port 22.  Required
user string The name of the user that we should use when connecting over SSH. It can be sensible to setup a new user specifically for deploy.dog.  Required
group string The group that this server belongs to. All servers must be a part of one group. You can then define in your deploy config how to handle the different server groups. E.g. you might run additional commands on your web servers to restart Nginx/Apache after a deployment.  Required
stage string All servers belong to a stage. You can name your stages anything you like, but typically people have stages like staging or production.  Required
type string This field defines the type of server that this is. At this time we only support servers that we can connect to via SSH, but we plan to add AWS S3 and other in the future.  Required
sshKey string This or a password is required for all SSH servers. This is the private key to use for our SSH connection. We highly recommend the use of a deploy.dog secret here, rather than including the key itself in your config file. You can generate SSH key pair secrets directly from your project page on our website, or create them manually.  Required conditionally
password string This or an sshKey is required for all SSH servers. This is the password to use for our SSH connection. We highly recommend the use of a deploy.dog secret here, rather than including the password itself in your config file. We recommend using sshKey over password where possible.  Required conditionally

Tasks Config

The next step is to define your tasks. Tasks are single commands (or groups of commands) that you need to run at some point. Tasks can be run at the build time, the deploy time and even the rollback time. You can re-use tasks in different stages of the process and even share them across server stages (so that you know what happens on staging always matches that of production, for example).

By default tasks run on the group of servers that you set to run them (as you'd expect), but there is an advanced feature of tasks that allows them on a different group of servers. This is setup using the delegateToGroup parameter. This can be used to do things like remove servers from a load balancer before they get code changes and then put them back in after. See below for more on this.

tasks:
  - name: build
    commands:
      - composer install
      - npm install
      - gulp build

  - name: phpunit_tests
    commands:
      - phpunit --configuration phpunit.xml.dist --coverage-text --colors=never --debug

  - name: remove_from_lb
    commands:
      - ./haproxyctl "disable server static/nginx_web1"
    delegateToGroup: lb

  - name: add_to_lb
    commands:
      - ./haproxyctl "enable server static/nginx_web1"
    delegateToGroup: lb

  - name: reload_nginx
    commands:
      - sudo service nginx reload

  - name: reload_phpfpm
    commands:
      - sudo service php7.0-fpm reload

  - name: migrate_db
    commands:
      - php app/console doctrine:migrations:migrate
    runOnce: true
{
  "tasks": [
    {
      "name": "build",
      "commands": [
        "composer install",
        "npm install",
        "gulp build"
      ]
    },
    {
      "name": "phpunit_tests",
      "commands": [
        "phpunit --configuration phpunit.xml.dist --coverage-text --colors=never --debug"
      ]
    },
    {
      "name": "remove_from_lb",
      "commands": [
        "./haproxyctl \"disable server static/nginx_web1\""
      ],
      "delegateToGroup": "lb"
    },
    {
      "name": "add_to_lb",
      "commands": [
        "./haproxyctl \"enable server static/nginx_web1\""
      ],
      "delegateToGroup": "lb"
    },
    {
      "name": "reload_nginx",
      "commands": [
        "sudo service nginx reload"
      ]
    },
    {
      "name": "reload_phpfpm",
      "commands": [
        "sudo service php7.0-fpm reload"
      ]
    },
    {
      "name": "migrate_db",
      "commands": [
        "php app/console doctrine:migrations:migrate"
      ],
      "runOnce": true
    }
  ]
}
Parameter Type/Value Notes Required
name string Your reference for this server. It will be show in logs, error messages, etc.  Required
commands array of strings The commands that you want to run. See the example above.  Required
delegateToGroup string This will get a different group of servers to run the task than the server you specify. Useful for getting another group of servers to do something on behalf of the server being deployed to at the time (such as removing it from a load balancer before the deploy and adding it back in again after).  Optional
runOnce boolean This ensures that the task runs just once (on the first server) even if you ask a group of servers to perform the task. Useful for things like database migrations (can be used in conjunction with delegateToGroup). When used together with serial in your Deploy Config, tasks will be run on one host in each serial batch (not just once in total). Defaults to false.  Optional

There is a 20 minute time limit on any individual command to prevent stalled commands from causing problems. If this is an issue, you should look to break up the work into multiple smaller commands that can complete within the time limit. Note, that the limit is per command, not per task. If this is still an issue, please get in touch with us explain why and we'll help you find a solution.


Build Config

We can build your project for you ready for deployment within an isolated docker container. You can use any publicly available docker image. The build works by running as many (or as few) tasks as you just defined above. The task's commands run one-by-one sequentially in the root directory of your project, within the docker container.

To help save time you can define directories for deploy.dog to cache after a build and resurrect at the beginning of each build. A good use-case for this is your vendor, node_modules, or similar directory with third party libraries in it.

buildConfig:
  - stage: production
    dockerImage: violuke/docker-php-project-for-ci
    tasks:
      - build
      - phpunit_tests
    cacheDirectories:
      - vendor
{
  "buildConfig": [
    {
      "stage": "production",
      "dockerImage": "violuke/docker-php-project-for-ci",
      "tasks": [
        "build",
        "phpunit_tests"
      ],
      "cacheDirectories": [
        "vendor"
      ]
    }
  ]
}
Parameter Type/Value Notes Required
stage string The build stage that this applies to. This allows you to have different build processes between test, staging and production (or whatever you call your stages) if required.  Required
dockerImage string The name of the publicly available docker image which you will use to build your project in. We recommend taking a look at some of our provided images as they may be helpful and your deploys may be faster as our servers are more likely to have the images cached already.  Required
tasks array of strings Set the names of the tasks which you have defined in the Tasks section. They will be run sequentially.  Required
cacheDirectories array of strings You can optionally provide a list of directories (relative to the project root) for deploy.dog to cache after a build and resurrect at the beginning of each build. They are specific to the stage. Cache directories not reused for a month will be deleted. At any time we cannot guarantee that a cache directory will be remembered from before. They should be a help for faster deploys, but not relied upon always being available. A good use-case for this is your vendor, node_modules, or similar directory with third party libraries in it.  Optional

Deploy Config

This is where you define your deployments. Deployments are defined per stage (e.g. production) and per server group (e.g. web or database). There are loads of options, so make sure you have a good read though the possible options below.

deployConfig:
  - stage: production
    group: web
    excludeFiles:
      #- .deploy.dog # This is excluded by default
      #- .deploy.dog.yml # This is excluded by default
      #- .deploy.dog.json # This is excluded by default
      - phpunit.xml
    oldReleasesToKeep: 8 # Defaults to 5 if omitted
    directory: /var/www
    copyFiles:
      - file: test1.json
        destination: SomeSubDir/test1-moved.json
      - file: test2.json
        destination: test2.json
    preUploadTasks:
      - remove_from_lb
    postUploadTasks:
      - upgrade_db
    postSymlinkChangeTasks:
      - reload_phpfpm
      - reload_nginx
      - add_to_lb
    preWebHook: http://requestb.in/whqqaiwh
    postWebHook: http://httpbin.org/post
    serial: 100%
    notifications:
      - type: hipchat
        onSuccess: true
        onFailure: true
        apiToken: c3ca4c19e9372735202431b41b25cd
        roomName: Dev Team
      - type: slack
        onSuccess: true
        onFailure: true
        url: https://hooks.slack.com/services/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
        channel: "#dev-team"
{
  "deployConfig": [
    {
      "stage": "production",
      "group": "web",
      "excludeFiles": [
        "phpunit.xml"
      ],
      "oldReleasesToKeep": 8,
      "directory": "/var/www",
      "copyFiles": [
        {
          "file": "test1.json",
          "destination": "SomeSubDir/test1-moved.json"
        },
        {
          "file": "test2.json",
          "destination": "test2.json"
        }
      ],
      "preUploadTasks": [
        "remove_from_lb"
      ],
      "postUploadTasks": [
        "upgrade_db"
      ],
      "postSymlinkChangeTasks": [
        "reload_phpfpm",
        "reload_nginx",
        "add_to_lb"
      ],
      "preWebHook": "http://requestb.in/whqqaiwh",
      "postWebHook": "http://httpbin.org/post",
      "serial": "100%",
      "notifications": [
        {
          "type": "hipchat",
          "onSuccess": true,
          "onFailure": true,
          "apiToken": "c3ca4c19e9372735202431b41b25cd",
          "roomName": "Dev Team"
        },
        {
          "type": "slack",
          "onSuccess": true,
          "onFailure": true,
          "url": "https://hooks.slack.com/services/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",
          "channel": "#dev-team"
        }
      ]
    }
  ]
}
Parameter Type/Value Notes Required
stage string The stage where this config applies. If you choose to deploy to production for example then all deployConfig values with a stage of production will run.  Required
group string The group of servers that this applies to. Allows for handling a different process on different groups of servers, such as you web, app and database servers.  Required
excludeFiles array of strings An array of the files/directories to exclude from the deployment. The .deploy.dog directory (if you have one) and your deploy.dog config file are automatically added to any files you specify.
  • /dir/ means exclude the root folder /dir
  • /dir/* means get the root folder /dir but not the contents
  • dir/ means exclude any folder anywhere where the name contains dir/
 Optional
oldReleasesToKeep integer Define how many previous releases you want to keep present on the servers for an fast rollback. Defaults to 5.  Optional
directory string The directory to deploy to. We will create a directory structure below this (see details). The root of your repository will reside in [directory-you-specify-here]/current (so this is often where you'll want to point your web server). Don't include a trailing slash.  Required
copyFiles array of objects Great for config/environment files. These files will be copied (with any deploy.dog Secrets converted to their real value) to your project. For each define the file to copy and the destination that it should be copied to.
  • file paths should be relative to [projectRoot]/.deploy.dog/ directory.
  • destination paths should be relative to your project's root directory. Be sure to always include the file name, not just destination directory.
 Optional
preUploadTasks array of strings Define the deploy.dog Tasks which you need to run on each of the servers referred to by the stage/group combination above prior to the deploy starting. The code will not be on the servers at this stage.  Optional
postUploadTasks array of strings As per preUploadTasks, except that the code has already been uploaded to your server (this runs before the symlink is changed).  Optional
postSymlinkChangeTasks array of strings As per postUploadTasks, except that the symlink will now be pointing to the new release directory.  Optional
rollbackPreSymlinkChangeTasks array of strings As per postSymlinkChangeTasks, except that executed in a Rollback rather than a Deploy.  Optional
rollbackPostSymlinkChangeTasks array of strings As per postSymlinkChangeTasks, except that executed in a Rollback rather than a Deploy.  Optional
preWebHook string We will post some JSON to this URL just before starting the deploy (after the build was completed successfully). We will post as many times as needed in order to satisfy deploying as per your serial setting. Example of JSON posted:

{
    "provider": "deploy.dog",
    "process": "deploy",
    "type": "pre",
    "timestamp": "2018-12-08T23:12:06+01:00",
    "stage": "production",
    "success": true
}
You can work on the assumption that nothing will be removed from this, but new values may be added in the future. Real payload will not be pretty-printed.
 Optional
postWebHook string We will post some JSON to this URL just after the deploy (on success or failure). This works in the same way as preWebHook, so see above for details. Example of JSON posted:

{
    "provider": "deploy.dog",
    "process": "deploy",
    "type": "post",
    "timestamp": "2018-12-08T23:10:41+01:00",
    "stage": "production",
    "success": true
}
You can work on the assumption that nothing will be removed from this, but new values may be added in the future. Real payload will not be pretty-printed.
 Optional
serial integer, string or array How many servers to deploy to concurrently. Can be an integer for the number of servers to do per run, or a percentage to determine value automatically based on group size. Can also be an array of values (integers and/or percentages). Defaults to 100% if omitted.  Optional
notifications array of objects Describes how would you like to be notified about the completion (successfully or otherwise) of your deployments. See the deploy.dog notifications documentation for full details on the different integrations.  Optional

Rollbacks

Rollbacks don't have their own configuration. They run effectively the same process as a deploy with the following exceptions:

  • There is no Build stage in the process as the code is already built (from when you originally deployed that version).
  • There is no uploading of code (as it's already on your servers).

Rollbacks are therefore typically much faster (as typically code retrieval, building and uploading are the slowest parts of a deployment process). Your Deploy config's rollbackPreSymlinkChangeTasks and rollbackPostSymlinkChangeTasks will be run instead of the other lists of tasks which run on deploy. Your notifications will be run just as if they were part of a normal deployment too and your serial setting will also be observed.


Variables

deploy.dog makes certain values available to you in variables which you can use within your config. These can be used in your Tasks or directly in your Build and/or Deploy config.

The possible variables are show below. To use a variable, you simply surround its name with double curly brackets and a space, like {{ variable_name }}. You may want to use them in your database migration scripts, webhooks, etc

Parameter Type/Value Notes Context
deploydog_release_version string The "Release Version" given to this release by deploy.dog. This will unique for all deployments (on a per project basis). It will also be the name of the directory within the releases directory of the deploy (see directory structure section below).
  • Deploy
  • Rollback
deploydog_commit_ref string The full Git commit reference for the version of the code being deployed, e.g. f152f23c567df2f2e10c698214a62e8621737d0c.
  • Build
  • copyFiles
  • Deploy
  • Rollback
deploydog_release_path string The full path to the directory that we are deploying to, e.g. /var/www/your-site/releases/20171905125323-123 (note, no trailing slash). If you need this, we recommend using this variable rather than building it yourself based on the deploydog_release_version to ensure future compatibility. See directory structure section below.
  • Deploy
  • Rollback
deploydog_releases_path string The full path to the directory where all releases are stored, e.g. /var/www/your-site/releases (note, no trailing slash). See directory structure section below.
  • Deploy
  • Rollback
deploydog_shared_path string The full path to the "shared" directory (where common releases assets can be stored), e.g. /var/www/your-site/shared (note, no trailing slash). See directory structure section below.
  • Deploy
  • Rollback
deploydog_is_rollback integer This will be a 0 in deployments and 1 in rollbacks.
  • Deploy
  • Rollback
deploydog_committer_email string The email address of the committer as per your Git repository.
  • Build
  • copyFiles
  • Deploy
  • Rollback
deploydog_committer_name string The name of the committer as per your Git repository.
  • Build
  • copyFiles
  • Deploy
  • Rollback
deploydog_deployer_real_name string The real name of the user who triggered the deployment, according to the data entered in their deploy.dog account.
  • Build
  • copyFiles
  • Deploy
  • Rollback
deploydog_deployer_username string The deploy.dog username of the user who triggered the deployment.
  • Build
  • copyFiles
  • Deploy
  • Rollback
deploydog_deployer_email string The email of the user who triggered the deployment, according to the data entered in their deploy.dog account.
  • Build
  • copyFiles
  • Deploy
  • Rollback
deploydog_server_name string The name of the server which is being deployed (as per your Servers config section).
  • Deploy
  • Rollback
deploydog_stage string The name of the stage that is being deployed (e.g. production or staging). This is the same stage that would lead to the selection of the correct servers, etc.
  • Build
  • copyFiles
  • Deploy
  • Rollback

Custom Variables

You can also define your own custom variables. These are particularly useful for open source projects. For example an open source project might be defining a configuration setting in an environment file (such as .env) and allowing users to set this value from their own config overrides. An example of a custom variable would be {{ myproject_cache_driver }}. Users deploying the project can then add the variable myproject_cache_driver to their Config Overrides so that they can use the project's deploy.dog.yml file, but make their own adjustments so it suits their needs.

With custom variables, you can also set a default value, so that if a project doesn't have it defined, this value would be used. A good example would be our myproject_cache_driver variable from above. The project would like to suggest the sensible default value of redis to make it easier for its users to configure the deployment.

  • Custom variable with no default value: {{ myproject_cache_driver }}
  • Custom variable with default value of redis: {{ myproject_cache_driver::"redis" }}

It should be noted that custom variable names can only consist of numbers 0-9, letters A-Z a-z, dashes - and underscores _.

If you need to include a double-quote character in your custom variable default value, you need to escape it with a backslash.

variables:
  myproject_cache_driver: redis
  why_we_are_here: unknown
{
  "variables": {
    "myproject_cache_driver": "redis",
    "why_we_are_here": "unknown"
  }
}

Directory Structure

In the directory that you specify in your Deploy config, we will create the following directory structure. You will typically want to point your web server to the current symlink which will always point to the root of your code base for the currently deployed version.

  • /var/www/your-site (as you defined in your Deploy config directory parameter)
    • current → /var/www/your-site/releases/20170804083212-789 (symlink to current release)
    • releases
      • 20170803150741-123 (two releases ago)
      • 20170803171452-456 (previous release)
      • 20170804083212-789 (current release)
    • shared (you can store your shared assets here, e.g. file uploads)

Example of a (Simple) Complete Config File

As you've just seen deploy.dog is really powerful and has loads of options. Don't get scared though, most projects only need the basics. Below is an example file for the deployment of a simple PHP application running on a single server with Slack notifications.

servers:
  - name: web1
    host: web1.example.com
    port: 22
    user: deploydog
    group: web
    stage: production
    type: ssh
    sshKey: (d.d.secret:web1-ssh-key)

tasks:
  - name: build
    commands:
      - composer install

buildConfig:
  - stage: production
    dockerImage: violuke/docker-php-project-for-ci
    tasks:
      - build
    cacheDirectories:
      - vendor

deployConfig:
  - stage: production
    group: web
    directory: /var/www
    notifications:
      - type: slack
        url: https://hooks.slack.com/services/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
        channel: "#dev-team"
{
  "servers": [
    {
      "name": "web1",
      "host": "web1.example.com",
      "port": 22,
      "user": "deploydog",
      "group": "web",
      "stage": "production",
      "type": "ssh",
      "sshKey": "(d.d.secret:web1-ssh-key)"
    }
  ],
  "tasks": [
    {
      "name": "build",
      "commands": [
        "composer install"
      ]
    }
  ],
  "buildConfig": [
    {
      "stage": "production",
      "dockerImage": "violuke/docker-php-project-for-ci",
      "tasks": [
        "build"
      ],
      "cacheDirectories": [
        "vendor"
      ]
    }
  ],
  "deployConfig": [
    {
      "stage": "production",
      "group": "web",
      "directory": "/var/www",
      "notifications": [
        {
          "type": "slack",
          "url": "https://hooks.slack.com/services/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",
          "channel": "#dev-team"
        }
      ]
    }
  ]
}