Projects Status Update – April 2017

I unfortunately procrastinated the March status update so much that it has melded into the April update. Over the past 2 months I’ve been working on 4 of my 10 Projects for 2017. Those 4 are: Paper Towers, Spanish, Krav Maga, and learning the Cello. My current core focus is learning the Cello and I’m spending 3 – 4 hours a week on the rest.

Paper Towers (Now Tower Storm again)

Firstly I decided to stick with the name Tower Storm instead of re-branding this project as Paper Towers. I’m proud to say that Tower Storm is finally Open Source!. I spent around 30 hours in the last 2 months removing all copyrighted graphics and sounds from the game, and cleaning up the code so it’s easy for anyone to run and make modifications. I finally pulled the trigger a few hours ago and put it up on Github. I also did my first open source development stream, if you’re interested in seeing how I develop Tower Storm, how to mod it yourself, and my thoughts on programming in general you can follow me on Twitch.

Spanish

As of the 21st of April I was up to a 25 day streak on Duolingo doing at least 30 minutes of spanish a day (50xp per day). Unfortunately I lost that streak, but I will be continually learning Spanish on Duolingo over the next year. Alongside that I’ve been watching movies, LoL games and random Youtube videos in Spanish. I’ve also switched my PC and Phone to use Spanish as their default language. Every little bit of learning helps.

Krav Maga

I signed up for Krav Maga 2 weeks ago and so far have done 6 classes. I’ll probably continue this 3 times a week schedule. So far we’ve practiced basic punching, hammer fist, kicking, roundhouse kicks, getting out of chokes, and defending yourself when you’re on the ground. I’m really enjoying it so far, it’s useful, it’s a ton of fun fighting as hard as you can (into pads) and it’s a great workout. I hadn’t been punched in maybe 15 years and it’s an eye opening experience, and made me realize I have to toughen up a lot if I want to survive a real life fight.

Cello

To celebrate the open sourcing of Tower Storm and the down-winding of that project I went out and hired a Cello for 3 months. I’ve only had it a week now and haven’t had lessons yet. I bought a beginners book and have been teaching myself how to play with that and YouTube tutorials. I’m excited to report that I can already play the beginning of the Rains Of Castamere song! Albeit with a lot of squeaks and quite slowly. I can’t wait to learn more.

So that’s it. I’ll be continuing to focus on these 4 activities until at least the end of May whereupon I may start on another depending on my progress and time available.

 

Hundreds of robots in every home

I used to have hundreds dreams and ideas I wanted to pursue. So many that I jumped from task to task like mad trying to make something happen and only ended up scratching the surface of a few. Recently I’ve realized there only 2 major ideas I keep coming back to that I wish to pursue more than anything else:

  • Building Tower Storm into a big successful game that millions of people play and enjoy and I earn enough off to never have to worry about money again.

  • Building a robotics company that makes producing your own food automatically a possibility for everyone on the planet.

Tower Storm is already in progress, so in this post I want to talk about my vision for the future of robots.

I don’t believe there will be human like robots in every home as in bionical man, but I do think there will be hundreds of machines that will do most menial chores automatically. They’re already being made, in a crude unrecognisable form, via 3D Printers and open source circuitboards.

This reminds me of computers in the 70’s, they were toys that you had to solder and program yourself, there was no way everyone was going to have one let alone use it all the time. I feel that robots are at a similar stage.

What is needed now is a standard platform that all robots can build upon to work together to accomplish larger tasks. The robots would be modular and accomplish one function in a simple way on their own and they could then be combined in thousands of ways or sold in pre-packed sets to the non geeks out there.

Lets take cooking for an example. Making a spaghetti is somewhat complicated, but each step and microstep is pretty simple. Lets break down the steps:

  • Cook Sauce
    • Put pot on hotplate
    • Turn hotplate on
    • Cook Onions
      • Peel onion
      • Slice onion
      • Transfer onion to pot
      • Stir for 2 minutes
    • Add Mince
      • Defrost mince
      • Transfer to pot
      • Stir for 2 minutes
    • Add jar of sauce
      • Open jar
      • Pour jar into pot
    • Stir for 10 Minutes
  • Cook Pasta
    • Open pasta packet
  • Put sauce on pasta
  • Add cheese on sauce

When you lay it out like that you can see there are a lot of small steps, but none are too complicated for a robot to do. So why don’t they do it? Well I believe they will 10 – 20 years from now, we simply need to build a system for robots to work together in a neat way (like unix pipes) so they can collaborate and make tasks like preparing dinner completely automated.

We’ve already got the infrastructure with 3D Printers and arduinos to make this happen, if the community created open source designs for robots that can do each of these tasks then anyone, anywhere could build robots that automate many of their cooking tasks for them, eventually not needing to cook at all.

What has me even more excited than cooking is producing food automatically. It seems even more complicated than cooking, but if we break it down into separate components we could build robots that create and maintain tiny at home vegetable farms or fruit trees.

Then if these designs are open sourced anyone anywhere in the world can live effortlessly off the grid with all their food made for them by a team of robots. Once we’ve automated much of the first worlds food supply we can even help out those less fortunate in the world. Everyone in the world could use them to have their own completely automated unlimited production farm.

This is what excites me most about 3D Printing and open source circuitboards. Not gadgets and toys but machines that can bring complete automation to everything we don’t want to do in our lives. so everyone in the world is free to focus on what they enjoy, creating amazing things and learning and giving back to society.

Sure it’s insanely complicated, but it’s doable,

 

Tower Storm and my 30 second 2 button deployment process

2 years ago I read an amazing book The Lean Startup which made me completely rethink how I developed applications and websites. In the past I’d make a change to an application, spend 30 minutes going through the main features and release. Inevitably a few days later customers would come back saying x, y and z are all broken. So I’d fix them and re-release and now 3 other completely different features would be broken. This happened consistently but with limited time and money I thought it was impossible to improve this process.

Today I can make a change to Tower Storm and within 30 seconds have this change live online with very little (soon to be none) manual testing and no old bugs are coming back to bite me. In this post I want to show you how I’ve done it.

Automated Testing

The first step is to eliminate this 30+ minutes of testing that I had to manually do after every change. There is absolutely no way you can quickly release and iterate on your app without either:

a. An army of testers who are always able to detect any bug and will happily retest everything in your application upon every single change
b. Automated tests that find regressions and issues in your application for you

Being that you’re running a lean startup here I don’t think you’ve got thousands of dollars to burn on many dedicated testers so lets explore the world of automated testing.

Automated testing is where you create code that tests the functions of your application to determine if it’s doing what it should be doing. For example if you have a function that should remove every comma from a string and you want to ensure that it works for a variety of different strings you might create a test like so:

function testRemoveComma() {
var sentence = "this,is,a,sentence";
var expectedSentence = "thisisasentence";
var actualSentence = removeComma(sentence);
assert.equals(expectedSentence, actualSentence);

}

In this javascript example we first create a sentence with commas in it, then we specify what we expect back from our function. Then we call that function and ensure that what we got back from it matches what we expect back from it.

This example is what is known as a “unit test”, it is a test that checks one function by giving it inputs and receiving outputs and it doesn’t do other tasks such as connecting to your database or reading files. It should only check one function only. If that function calls other functions you need to use a technique called ‘mocking‘ so that they don’t really get called. I’ll go into more detail on how to create unit tests and mock objects in a variety of languages in a later post.

To start unit testing you’ll need a library to run these tests. Generally there is one or two good testing libraries for most languages. For Javascript I’d recommend mocha for Node.js testing or Jasmine for client side testing, for Java JUnit with Mockito is awesome and for PHP PHPUnit works well.

Unit tests are the simplest, fastest and often most fragile tests. They aren’t the best for ensuring your app is bug free but they are perfect for eliminating existing bugs and ensuring they never occur again.

The thing I love about unit tests is because they are so fast and easy to write you can do a process known as test driven development. This is where you write unit tests for your code before you write a single line of code. So in the remove comma example above we could write an empty removeComma function, then write the above test and run it only to see it fail, then after it has failed we create our removeComma function and run the test again and when it passes it means our code is working.

When you do test driven development constantly you can save hours by not needing to run your app after each code change. You simply test then code, test then code and eventually at the end you run your app and because every function works as it should your app should (in theory) work perfectly first go. It’s amazing when you get into this flow because if you’re building a large app you can continue to code for hours and really get into the zone without having to recompile everything and boot up your app to see if it’s working as it should.

Better testing with Integration and Acceptance tests

After you’ve mastered the art of the unit test there are even more powerful tests that you can use that will allow you to deploy your application without even running it and know that all functionality is working.

You do this by creating integration and acceptance tests. Unlike unit tests Integration and Acceptance tests actually test your app in a real environment with database and network calls. Integration tests are similar to unit tests in that they run one function or a group of functions in order and check that they are all working as they should. The difference is integration tests run the code as if a real user was calling the function, so if the function creates records in the database the integration test will do that, and if your function calls another external service the integration test will do that too.

Here’s an example of a PHP integration test in ZenTester:

/**
* @test
*/
function Checklogin() {
$this->ci->load->library('userManager');

$random_key = self::$random_key;
$reg_data = ut_get_registration_data($random_key);

//logout first (check_login function in controller does this too). 
$this->assertEquals($this->ci->user->logout(), true, "logging out before logging in");        
$this->assertEquals(is_array($this->ci->usermanager->register_user($reg_data, 0)), true, "Registering User With All information");
$this->assertEquals($this->ci->user->login($reg_data['email'], $reg_data['password']), true, "logging in to this user");
$user_id = $this->ci->user->get_user_id();
$this->assertEquals($this->ci->user->is_valid_user(), true, "Checking that after login we are a valid user.");
$this->assertEquals($this->ci->user->logout(), true, "Testing logging out");
$this->assertEquals($this->ci->user->is_valid_user(), false, "Checking that we are not a valid user after logging out. ");

ut_erase_user($user_id);
}

In this integration test we first create a new user with the helper function ut_get_registration_data. Then we register and log in with that user. After logging in we ensure that the user has successfully logged in and is valid. Then we log out and check that this also worked. Finally the user is deleted at the end.

In this case we create and clean up all our data so the database isn’t littered with test data. The downside of always deleting your data at the end of the test is you may find that it’s hard to track down why an integration test is failing because you can’t see what was created and what wasn’t. At Wotif we don’t clean up our data at the end of each tests and instead re-use the test data upon every run and delete old data at the beginning of each test. This way you don’t add much test data to the database while still being able to figure out what went wrong when a test fails.

Acceptance tests are another level of abstraction, they use your app from a users perspective, loading pages, clicking on links etc and ensuring what is shown to the user after performing specific functions is correct. They are often done with tools such as selenium or curl. At Wotif we’ve been using CucumberJVM to run selenium on a remote box which loads up our app, tests that all the main features are working from a user perspective and reports if anything is broken. These are then run automatically by Team City every time we push a change.

 

Using GruntJS to build your assets 

Grunt is the second most amazing part of the deployment process. It basically takes the application and builds it so it’s ready to upload. It currently does all of the following (the grunt plugin used to do each item is in brackets):

  • Bumps the game version number (grunt-bump)
  • Checks the javascript for any errors (lint)
  • Cleans the build directory (grunt-contrib-clean)
  • Copies all the correct files to the build directory (grunt-contrib-copy)
  • Turns all the cofeescript into javascript (grunt-contrib-coffee)
  • Builds the core game javascript into a single js file (grunt-exec which runs impact.js’s build tool which is coded in php)
  • Uglify’s (minifies) the javascript along with all external libraries and config files into a single game.min.js file (grunt-contrib-uglify)
  • Compiles all the less css files (grunt-contrib-less)
  • Minifies the css and puts it into one file (grunt-contrib-cssmin)
  • Compiles the jade files into html (grunt-contrib-jade)
  • Uploads all the compressed and compiled assets to a new folder on amazon s3, the folder name is the current game version number (grunt-s3)

It’s a complicated process yet grunt.js handles most of these tasks with very little configuration needed and can do all of this in under 30 seconds.

The assets are uploaded to a new amazon s3 folder of the builds version number so that assets are never overwritten and users who are still playing the game are not interrupted. You can do this by setting the variable pkg to your package.json file then using the variable <% pkg.version %> in your s3 upload script. My s3 task looks like this:

grunt.initConfig({
bump: {},
pkg: grunt.file.readJSON('package.json'), 
s3: {
bucket: 'towerstorm',
access: 'public-read',

// Files to be uploaded.
upload: [        
{
src: 'build/public/js/lobby.min.js',
dest: 'game-server/<%= pkg.version %>/js/lobby.min.js',
gzip: true
},
{
src: 'build/public/css/lobby.min.css',
dest: 'game-server/<%= pkg.version %>/css/lobby.min.css',
gzip: true
}
]
}
});

If you’re using grunt-bump to auto bump the version number with every build you’ll also need to modify the grunt-bump/tasks/bump.js file and add the following line to the bottom of the grunt.registerTask function so that after the version is bumped the variable pkg is set to the latest version:

grunt.config.set("pkg", grunt.file.readJSON("package.json"));

In the game code it simply loads the assets for it’s current version number so even if people start games after this build process is done they will load the old game assets and it’s only when the new version is deployed and Node.js is restarted that the new assets will be loaded. This way the server code and game client code are always in sync. Lastly versioning the assets also ensures that users browser don’t cache old assets which could cause errors if gameplay changes are introduced yet clients are loading an old cached version of the game.

All the TowerStorm servers are hosted using Amazon EC2 and in the future I’m looking to implement a system where with each new version a bunch of new servers are spawned with the new game client and assets, then whenever players start new games they are all started on the new servers only and the old servers only stay alive until the last game is finished then they are powered down. This will allow us to continually release new versions of Tower Storm without ever having ‘patch downtime’.

Continuous Integration

The third step is to take this unit testing and asset building and automate it with a dedicated server that runs everything in a server like environment. This way if you have a team of developers they don’t each have to set up grunt and acceptance tests and full build environment on their machine, instead every time they commit a new change the continuous integration server downloads the new code from git, compiles it using grunt and runs all the unit tests using either a custom private server setup or running them on it’s own machine using it’s own browser or a headless browser like phantomjs.

I haven’t yet set up a continuous integration server for Tower Storm as I’m currently the only developer and it was easier to set everything up locally (especially in these very early stages) but I’ll definitely be setting on up soon. At Wotif we’ve tried out Jenkins, Bamboo and Teamcity and all were good in some ways and bad in others. I myself prefer the layout and feel of Bamboo the most however this is often personal preference as other members of our team prefer Teamcity’s layout more. Jenkins is probably the least liked in usability and layout but it is completely free and comes with tons of plugins for doing almost every task you like so if that’s what you’re looking for then it’ll work well for you.

Automated cmd files and the 2 button deploy process

To tie all these various testing, running and deploying scripts together I’ve created a few command files (yes I run windows 8, although I use Ubuntu at Wotif and the Tower Storm servers are running Linux) that make things even easier. Here’s what they do:

commitAndPush.cmd – Runs tortoisegit (my favourite git gui by far) commit screen then push’s the code after you’ve committed your changes. It looks like so:

call E:\\apps\\scripts\\tgit.cmd commit C:\\coding\\node\\towerstorm\\GameServer
call git push --all --progress  BitBucket
pause

the tgit.cmd file it refrences is a hook to tortoisegit to make it run any command from the command line. It’s contents are:

"C:\\Program Files\\TortoiseGit\\bin\\TortoiseGitProc.exe" /command:%1 /path:%2

devenv.cmd – Runs the game locally using node-dev which makes it auto restart whenever a change is made and it also runs test.cmd explained next:

set NODE_ENV=development
start cmd.exe /K "cd .. && call ./node_modules/.bin/node-dev server.coffee"
start cmd.exe /K "cd .. && call scripts/test unit"

test.cmd – This loads a cmd prompt that automatically runs all the unit tests using mocha and re-runs them whenever a test is made. It scans the test directory for all coffeescript files and runs them:

setlocal EnableDelayedExpansion
IF "%1" == "" (
SET files=C:\\cygwin\\bin\\find test -name "*.coffee"
) ELSE (
SET files=C:\\cygwin\\bin\\find test\\%1 -name "*.coffee"
)

FOR /F %%i IN (' %files% ') DO SET tests=!tests! %%i 
.\\node_modules\\.bin\\mocha --watch --reporter min --slow 10 --globals $ --compilers coffee:coffee-script --require coffee-script test\\_helper %tests%
pause

I run these scripts by binding them to the macro keys on my Logitech G15 keyboard (which I bought mainly because it had these keys). I have the dev environment setup bound to one key, grunt bound to another and commit and push bound to a third. This way I can develop in one key press and deploy a new version of Tower Storm using just 2 buttons 🙂

Hope this was informative enough and if you have any questions or are confused about any steps let me know.

 

Multiplayer online Tower Defense incoming :)

The past few months I’ve been working on a browser based multiplayer tower defense game called Tower Storm, because it’s something I’ve been wanting to build for years.

I had originally planned for it to be PvE where players work together to kill monsters similar to Azure Tower Defense, however we soon realised while PvE is easier for most people to understand it doesn’t have enough replayability without constantly adding new content.

So I’m focusing on PvP first in the style of Wintermaul Wars because I absolutely love it’s gameplay.

I’ve been play testing with friends on the home lan for a while, hopefully will have a version online soon.