How to drive your Ghost blog using Continuous Delivery - Part Three

Before we begin, this blog post is part of a 3-part series. You can read the all three blog posts here:

Part 1: How to drive a Ghost Blog using Continuous Delivery

Part 2: How to drive a Ghost Blog using Continuous Delivery

Part 3: How to drive a Ghost Blog using Continuous Delivery

In the first and second instalments of this series we learnt how to setup an AWS EC2 instance to host our blog, get started locally with Ghost and have an instance of Ghost locally that we can begin to develop with. If you're with me so far massive congrats. You've done well and I hope you've learned a bunch already.

I've got some even better news for you though. The best bits are yet to come!

Up Next

In this instalment we're going to finish everything off with a bang! We'll start with setting up a webhook in Github; this will allow you to notify your blog that there's been a change on the master branch of your blog. Thing is, for that to happen, you need an app of some kind listening to webhook requests from Github. When the app receives a request it'll kick-off some form of script (in our case a shell script) that will grab the latest source code, perform any sort of backups required, teardown the existing blog code, import the new one and pull the backup data back-in, finally the blog will be restarted. That's a fair bit of work so strap in and let's dig in.

Github Webhooks

Webhooks are powerful little creatures. They basically allow you to listen to (and act on) events that are happening in your source code repositories. To our ends we'll listen to all push events to and from our repository. This will involve add a webhook in Github for our repository so let's do that.

Adding a Push Webhook

Navigate to the Ghost repository you previously set up in the last instalment click 'Settings' followed by the 'Webhook & Services' link on the left-hand panel. Click the 'Add webhook' button. Fill in the 'Payload URL' field as follows:

http://yourdomain.com:4567/github-update

Leave Content Type as 'application/json' and enter 'thisismysecret' in the secret box. You'll use this secret as a password that will prevent anyone from just kicking off your blog any time they like (change this to something less easy to guess if you want just make sure to substitute it). Ensure the 'just the push event' option is highlighted and the checkbox 'active' is selected. You're good to add the webhook now - push the button!

Each POST request will contain a json payload that looks as follows, here's an example from my blog:

    {
      "ref": "refs/heads/master",
      "before": "034ba287262db13f2685bea5971e70b390444d5d",
      "after": "23751e02d24e344781b5c7ef2be294fc2e131bb7",
      "created": false,
      "deleted": false,
      "forced": false,
      "base_ref": null,
      "compare": "https://github.com/murphyj/devangst/compare/034ba287262d...23751e02d24e",
      "commits": [
        {
          "id": "f80d68a39840dac91f38ea3704e859af34326812",
          "distinct": true,
          "message": "Fixed devangst home for init script",
          "timestamp": "2015-02-22T18:12:24Z",
          "url": "https://github.com/murphyj/devangst/commit/f80d68a39840dac91f38ea3704e859af34326812",
          "author": {
            "name": "James Murphy",
            "email": "myemail@googlemail.com",
            "username": "murphyj"
          },
          "committer": {
            "name": "James Murphy",
            "email": "myemail@googlemail.com",
            "username": "murphyj"
          },
          "added": [

          ],
          "removed": [

          ],
          "modified": [
            "tools/init-devangst.sh"
          ]
        },
        {
          "id": "a87d2ec92cfb5f95f35f90c0dfca8cb9e81dd938",
          "distinct": true,
          "message": "Updated dev database",
          "timestamp": "2015-02-22T18:12:35Z",
          "url": "https://github.com/murphyj/devangst/commit/a87d2ec92cfb5f95f35f90c0dfca8cb9e81dd938",
          "author": {
            "name": "James Murphy",
            "email": "myemail@googlemail.com",
            "username": "murphyj"
          },
          "committer": {
            "name": "James Murphy",
            "email": "myemail@googlemail.com",
            "username": "murphyj"
          },
          "added": [

          ],
          "removed": [

          ],
          "modified": [
            "ghost/content/data/ghost-dev.db"
          ]
        },
        {
          "id": "63ba1ff61ca5e30fbff8f4918e2a39f2548ccca2",
          "distinct": true,
          "message": "Added css styling for disqus threads",
          "timestamp": "2015-02-22T18:12:50Z",
          "url": "https://github.com/murphyj/devangst/commit/63ba1ff61ca5e30fbff8f4918e2a39f2548ccca2",
          "author": {
            "name": "James Murphy",
            "email": "myemail@googlemail.com",
            "username": "murphyj"
          },
          "committer": {
            "name": "James Murphy",
            "email": "myemail@googlemail.com",
            "username": "murphyj"
          },
          "added": [

          ],
          "removed": [

          ],
          "modified": [
            "ghost/content/themes/casper/assets/css/screen.css"
          ]
        },
        {
          "id": "ed4b9c92703962752cdb675edad6046c54b8c238",
          "distinct": true,
          "message": "RELEASE It!",
          "timestamp": "2015-02-22T18:13:11Z",
          "url": "https://github.com/murphyj/devangst/commit/ed4b9c92703962752cdb675edad6046c54b8c238",
          "author": {
            "name": "James Murphy",
            "email": "myemail@googlemail.com",
            "username": "murphyj"
          },
          "committer": {
            "name": "James Murphy",
            "email": "myemail@googlemail.com",
            "username": "murphyj"
          },
          "added": [

          ],
          "removed": [

          ],
          "modified": [
            "ghost/archive/devangst-latest.zip"
          ]
        },
        {
          "id": "23751e02d24e344781b5c7ef2be294fc2e131bb7",
          "distinct": true,
          "message": "RELEASE",
          "timestamp": "2015-02-22T18:13:23Z",
          "url": "https://github.com/murphyj/devangst/commit/23751e02d24e344781b5c7ef2be294fc2e131bb7",
          "author": {
            "name": "James Murphy",
            "email": "myemail@googlemail.com",
            "username": "murphyj"
          },
          "committer": {
            "name": "James Murphy",
            "email": "myemail@googlemail.com",
            "username": "murphyj"
          },
          "added": [

          ],
          "removed": [

          ],
          "modified": [
            "ghost/archive/devangst-latest.zip"
          ]
        }
      ],
      "head_commit": {
        ...
      },
      "repository": {
        ...
      },
      "pusher": {
        "name": "murphyj",
        "email": "myemail@googlemail.com"
      },
      "sender": {
        ...
      }
    }

Accept webhooks

I performed a bit of investigation work here... I was hoping that there would be some pre-canned Node.js projects I could use but if I'm honest like everything Node.js related there were a bunch of bugs and things just didn't work. I tried (gith etc) that didn't work and it all felt like I was using a sledgehammer to crack a nut - is there any real need? We just need a really lightweight app that runs on a particular port accepting HTTP POST json payloads.

What sprang to mind? Sinatra and Ruby. Sinatra is really really lightweight and an excellent choice in my mind for this purpose. Ruby is very easy to install as well so it's the perfect compliment. We can set up Sinatra and ruby in two simple commands on Ubuntu 14.04:

    sudo apt-get install rbenv
    sudo gem install sinatra

Next we create a single file called webhooks.rb that will act as our Sinatra app for dealing with the webhook requests from Github (just one file - it did tell you it was pretty lightweight right?!):

require 'sinatra'  
require 'json'

post '/payload' do  
  payload_body = request.body.read
  verify_signature(payload_body)
  push = JSON.parse(payload_body)
  puts "I got some JSON: #{push.inspect}"
  result = %x[ ./webhook.sh ]
end

def verify_signature(payload_body)  
  signature = 'sha1=' + OpenSSL::HMAC.hexdigest(OpenSSL::Digest.new('sha1'), 'thisismysecret', payload_body)
  return halt 500, "Signatures didn't match!" unless Rack::Utils.secure_compare(signature, request.env['HTTP_X_HUB_SIGNATURE'])
end  

Generate a secure secret

If you want to make your secret ultra-secure right now trying running the following command at the terminal:

    ruby -rsecurerandom -e 'puts SecureRandom.hex(20)'

Replace 'thisismysecret' with the output of the above result to make your Sinatra app endpoint secure.

Put the files outside of your project (because you'll be tearing that down on your EC2 instance. I put them in the folder /var/www/webhooks/

Create a shell script for the dirty work

As you can see we've referenced the shell script webhook.sh that will sit in the same folder as the Sinatra webapp we just created. Our webhook.sh will perform the grunt work:

  • Pull down the latest zip file from Github using the v3 Github API
  • Unzip the site to a temporary location
  • Stop your existing Ghost blog (you must not back up the blog whilst Ghost is currently running according to the docs)
  • Stop Nginx
  • Backup existing data or images from our blog
  • Move over the new ghost blog
  • Reinstall new node modules
  • Copy back over data and images
  • Make sure permissions etc are restored
  • Restart Ghost
  • Restart Nginx
  • Try making a change to your blog and admire your handiwork

Generate a Secure Auth Token for Github

Prior to creating the script you need to generate an auth token for accessing the Github v3 API so that you don't need to use OAuth 2 (which is a big pain btw!). Just bear in mind to keep the auth token safe because it's effectively just password access to your Github account - so guard it closely. Log into Github and select the 'Settings' menu, choose the 'Applications' option in the left-hand side and click 'Generate new token' with a name you recognise using the default options then generate the token. Make a note of the password generated as you'll need it for the 'your_auth_token_from_github' part in webhook.sh.

This means that the webhook.sh file will look as follows (obviously substituting yourdomain with your actual domain and the auth_token; also comment out images lines if you have images (I don't yet)):

    #!/bin/bash


    # First, get the zip file
    echo "Retrieving zip file of yourdomain-latest from Github..."
    cd /tmp && curl -v -H 'Authorization: token <your_auth_token_from_github>' -H 'Accept: application/vnd.github.3.raw' -o yourdomain-latest.zip -L https://raw.github.com/user/yourdomain/master/ghost/archive/yourdomain-latest.zip

    # Second, unzip it, if the zip file exists
    if [ -f /tmp/yourdomain-latest.zip ]; then
      # Unzip the zip file

      mkdir -p /tmp/yourdomain-latest
      cd /tmp/yourdomain-latest
      echo "Unzipping yourdomain-latest..."
      unzip -q ../yourdomain-latest.zip

      # Delete zip file
      cd /tmp
      #rm yourdomain-latest.zip

      # Rename project directory to desired name
      mv yourdomain-latest yourdomain.com

      # Stop Ghost -- so we can back up
      sudo ~/npm/bin/pm2 stop ghost
      sleep 2

      # Create a local backup of images and data
      echo "Backup ghost database to /tmp/yourdomain-data-backup"
      mkdir -p /tmp/yourdomain-content-backup/data
      sudo cp -R /var/www/yourdomain.com/ghost/content/data/* /tmp/yourdomain-content-backup/data

      #mkdir -p /tmp/yourdomain-content-backup/images
      #sudo cp -R /var/www/yourdomain.com/ghost/content/images/* /tmp/yourdomain-content-backup/images

      # Delete current directory
      echo "Removing old yourdomain.com site from /var/www/devangst.com"
      rm -rf /var/www/yourdomain.com

      # Replace with new files
      mv yourdomain.com /var/www/

      # Perhaps call any other scripts you need to rebuild assets here
      # or set owner/permissions
      # or confirm that the old site was replaced correctly
      cd /var/www/yourdomain.com

      # need to stop nginx first before you can install node_modules
      echo "Stopping nginx..."
      sudo service nginx stop
      echo "Waiting for nginx to start..."
      sleep 2
      echo "Running npm install --production"
      cd /var/www/yourdomain.com/ghost
      sudo rm -rf node_modules
      npm install --production
      sleep 60

      echo "Move database files and images backed up back"
      sudo rm -rf /var/www/yourdomain.com/ghost/content/data
      #sudo rm -rf /var/www/yourdomain.com/ghost/content/images

      sudo mkdir -p /var/www/yourdomain.com/ghost/content/data
      #sudo mkdir -p /var/www/devangst.com/ghost/content/images

      sudo mv /tmp/yourdomain-content-backup/data/* /var/www/yourdomain.com/ghost/content/data
      #sudo mv /tmp/yourdomain-content-backup/images/* /var/www/yourdomain.com/ghost/content/images

      sudo rm -rf /tmp/yourdomain-content-backup

      sudo chown -R ubuntu:ubuntu /var/www/yourdomain.com

      echo "Starting ghost via pm2"
      sudo NODE_ENV=production ~/npm/bin/pm2 restart /var/www/yourdomain.com/ghost/index.js --name "ghost"
      echo "Starting nginx..."
      sudo service nginx start
    fi

Generate archive for your blog

Next you need to generate a zip archive file so you can pick up all the latest changes to your blog and send them over to your AWS account. To do this we create a simple shell script that we run 'pre-check in'. I called the file ziparchive.sh and placed it in the root folder of my project:

    git archive -o ghost/archive/yourdomain-latest.zip HEAD

You'll also want to make it executable:

    chmod +x ziparchive.sh

So on every check in to master you'll want to re-generate your archive otherwise no changes will be reflected on your site. Do this simply by running the shell script and checking the ghost/archive/yourdomain-latest.zip in when you commit to master:

    ./ziparchive.sh

Exclude *-latest.zip from your git archive

So to get around this you have to great a .gitattributes file in the root of your project containing:

.gitattributes export-ignore
.gitignore export-ignore
/ghost/archive/devangst-latest.zip

This will ensure that when the git archive command is run the zip file will be excluded from the archived zip.

Getting webhooks to work

There's a bit of a chicken and egg situation, in that, the first time you check-in your project you can't pull down the latest code. So what I've done is I manually created the webhook stuff and put it in /var/www/webhooks first then set it up to run. However, I also check-in the webhooks stuff into my project so it's similarly copied over to /var/www/yourdomain.com/webhooks. That way I can always have an updated copy of the webhooks I can manually copy over if I'm updating the build (we could automate this part as an improvement but that'll be out of scope for this post).

Once we have the files on our EC2 instance we simple run the following command to get the WEBrick server running our Sinatra app:

    nohup ruby webhooks.rb -e production >> /var/log/webhook-yourdomain.log 2>&1 &

This will start a production version of the webhooks using nohup. The & command will mean that we don't have to keep a terminal session open and it'll just run in the background for us.

Again we have a chicken and egg situation with the webhook.sh script too. It assumes that a lot of the code is already present (we could make it defensive but it's easy enough to manually copy and unzip the files the first time). Copy over the archive file you've created locally:

    scp -i /path/to/your-key.pem /path/to/repo/ghost/archive/yourdomain-latest.zip ubuntu@ec2-XX-XXX-XXX-XX.eu-west-1.compute.amazonaws.com:/var/www/yourdomain-latest.zip

Then jump onto the EC2 instance and unzip it:

    ssh -i /path/to/your-key.pem ubuntu@ec2-XX-XXX-XXX-XX.eu-west-1.compute.amazonaws.com
    cd /var/www
    mkdir -p yourdomain.com
    unzip ../yourdomain-latest.zip

Great! You now have a copy of your blog with all files up-to-date on AWS. We're ready to start the automation process. First though, let's check we can actually run a copy of ghost.

Run your Ghost blog

Before automation let's make sure our Ghost blog is accessible. We do this by running it using pm2:

    NODE_ENV=production ~/npm/bin/pm2 start /var/www/yourdomain.com/ghost/index.js --name "ghost"

It's entirely plausible there are some permission issues running pm2 so you can either opt to run it using sudo or try fixing permissions where you can:

    sudo chown -R ubuntu:ubuntu /var/www/yourdomain.com

I had some other issues with pm2 access as I'd previously tried running pm2 as root. E.g.

    Following error: EACCES
    sudo ~/npm/bin/pm2 kill && rm -rf ~/.pm2/

Another great benefit of pm2 is that it'll automatically reboot your process should pm2 fail for any reason:

Let's clear the processes first

    ~/npm/bin/pm2 dump

Then to ensure it starts up

    sudo ~/npm/bin/pm2 startup ubuntu

Ensure Nginx is running

Again in the last instalment I showed you how to set up Nginx on EC2. It should be running but let's restart it to be sure.

    sudo service nginx stop
    sudo service nginx start

Go to your blog at yourdomain.com in the browser and check to see if Ghost is working. Make sure this is working perfectly before you move to the next automation step because you need to know Ghost will run normally with Nginx before you even consider the automation aspect of it.

Checking automation

BACKUP, BACKUP, BACKUP: Please ensure that you back-up a copy of your ghost.db inside ghost/content/ghost.db if you already have existing posts that you care about or you could try this entirely with a ghost install that you don't care about. Either way I strongly suggest you back things up just to be sure. The shell script is pretty unforgiving if there are any typo's or issues and we wouldn't want to be losing those precious posts we spent all our time writing now would we?!

You're now in a good place (after backing everything - maybe back up all your site to a zip somewhere else and pull it down to your machine just to be sure you have it - EC2 instances have been known to die!). Try committing your yourdomain-latest.zip to master by running the ziparchive.sh script with some changes so you know it's working. Also, add some example posts to make sure the database back up is working correctly.

If you try it and it works - brilliant! I hope that 'bang' I spoke about earlier is what you're feeling right now and you see just how powerful all this effort was. The freedom to just check in your work whenever without having to bother about deployment to live is a liberating feeling!

Loose Ends

I hope this is a fairly comprehensive walkthrough but I may have missed something. If I have let me know and I'll change the docs (sorry!). This is unfortunately quite a complex sequence of changes so it's entirely possible I haven't gotten everything nailed down perfectly for you - I'm happy to help though you are having issues just drop me a note.

This isn't full-proof by any means. We still need a few more features to make this really powerful, however, I have a few ideas I might blog about in the next instalment to make it even better:

  • Only rebuild changes made to master branch
  • Ensure Ghost blog is always up (at present it will be unavailable whilst the site rebuilds)
  • Automatic daily backups
  • Persistent ghost.db in AWS rather than SQLite
  • Automatic sitemap.xml with robots.txt for SEO

If you have any suggestions let me know and I'll take a look.

Thanks for journeying with me this far - please follow me on my personal twitter @jmurphyuk or my @developerangst twitter account or you could even subscribe to this RSS feed too! Until next time...

James Murphy

Java dev by day, entrepreneur by night. James has 10+ years experience working with some of the largest UK businesses ranging from the BBC, to The Hut Group finally finding a home at Rentalcars.

Manchester, UK
comments powered by Disqus