Fixing node-gyp: Permission Denied when running as root

This node-gyp permission denied issue happens when, as the root user, you globally install node modules that require compilation with node-gyp. As root is the default user on Digital Ocean this seems to affect many people and should probably be fixed at the root cause.

Luckily there is a pretty simple workaround for this:

npm config set user 0

The other solution is to create a regular user and install everything as them. This is safer as installing modules as root means they could do anything to your system. I’d recommend installing all node modules as a normal user without root access if you’re on a machine with anything important on it.

 

Request: Buttons for logging life events for a quantified self

I love the idea of the quantified self, where you record your life inputs (exercise, sleep, food, supplements, etc) and outputs (happiness, focus, energy, etc) and draw correlations between them. My main barrier to this is the data entry itself – It’s hard to record events on a consistent basis. I’d prefer it to be either completely automatic or so easy that I don’t need to actively think about it. 

I’m seeking a device to track life events with a single button press. There are some solutions available, but I haven’t found any that are:

  • Easy to setup (no hardware hacking required)
  • Reasonably priced (<$10 per input button)
  • Hardware based (so that it’s right in front of me as the event happens, I don’t have to remember to open an app)

Solution

An ideal solution would be a series of buttons I can stick to the wall where I have something I want to track. Each button sends a request to an API endpoint with a payload of data related to that button. So for tracking my supplements I could have a button for “Protein Powder”, “Magnesium”, “Vitamin D” etc that I press as I take them, and they each record an event in say AWS CloudWatch Metrics. 

It could work this like: I buy this device, it is a raspberry pi or other cheap compute option with a USB port. I plug it in at home and it creates it’s own WiFi network that I can join on my laptop. It then has a simple website that I can go to in my browser. On this website I can:

  • Make it connect to my home WiFi network for outbound access
  • Select a service or API endpoint I would like it to send messages to
  • Bind key presses to different API payloads. 

I can plug a keyboard or other button like device into the USB port, and then in the web interface I can click “bind button”, press a button on the keyboard, and choose the payload that button will send. 

Out of the box it could be configured for say CloudWatch Metrics. You choose that you want to send your requests there, input your AWS access key, and then for each button you type a string for what it represents, like “Protein Powder”, or “Vitamin D”. It handles formatting the payload to what makes most sense for the service you’re sending the data to.  You should also be able to have a metric like “Happiness Level” and bind the numbers 1 -> 10 to send the “happiness” payload with the number you pressed as a dimension. 

It could get more providers over time, and you could create your own via a configuration file on the device. Then you can select your provider in the web interface and all the button payloads are automatically formatted correctly for that provider. 

You could place these devices anywhere you want to measure a life input or output: Beside your supplements, in your kitchen, at your work desk to measure productivity, at a doorway to mark your happiness as you enter and leave a room etc. 

The company could sell this device by itself, and could also sell a 4×4 grid or 10 x 1 row of buttons with spaces for labels as an add-on to make tracking even easier. 

This doesn’t seem too complicated to create, but I don’t have the time right now, and would like someone to scale this, building a company selling these to others. I think there’s a market for it and I’d like everyone to be able to build their own quantified lives and discover what makes them productive and happy. I would pay up to $100 for a device like this, maybe more for early access.

 

How to add if statements to your CloudFormation CDK templates

I recently came upon a situation where I was building an application using AWS CDK (Which is a package that allows you to write Typescript code that compiles to a CloudFormation template) and I needed to reference a bucket in the template, but the bucket name would be different in Beta versus in Prod. I naively thought I could just do this:

// Don't use this, this code doesn't work
const stage = new cdk.Parameter(this, "Stage", {
  default: "beta",
  type: "String"
}).value.toString();

const bucketName = stage === "prod" ? "app-prod-bucket" : "app-beta-bucket";
const Bucket = s3.Bucket.import(this, "AppBucket", {
  bucketName
});

The mistake I made is forgetting that CDK is compiled into CloudFormation templates, so when I run cdk synthesize it takes this code, evaluates it, sees that the variable stage is currently not set (because it’s provided at runtime), and compiles the string “app-beta-bucket” into the template.

Then when running this compiled template it doesn’t matter what the parameter Stage is, that if statement has already been compiled out and it’s always going to set the bucket name to “app-beta-bucket”.

To fix this you need to use CloudFormation Conditional Statements, like so:

const stage = new cdk.Parameter(this, "Stage", {
  default: "beta",
  type: "String"
}).value.toString();

new cdk.Condition(this, "UseProdBuckets", {
  expression: cdk.Fn.conditionEquals(stage, "prod")
});

const bucketName = cdk.Fn.conditionIf(
  "UseProdBuckets",
  "app-prod-bucket",
  "app-beta-bucket"
);

const Bucket = s3.Bucket.import(this, "AppBucket", {
  bucketName
});

Now your if statement will be compiled into the template and the bucket name will change depending on the value of the “Stage” parameter

 

Back to WordPress

I gave Ghost a good shot. And I do love it as a blogging platform, the markdown and writing experience is beautiful. But I’ve switched this site back to WordPress because I want it to be more than just a blog. I’ve been inspired by Derek Sivers’s site and want to make this a platform for many different videos, reviews, and projects I’m working on. If you notice anything broken please let me know.

I’ve also installed the Jetpack Markdown plugin because writing in markdown is so much nicer and cleaner than the default wordpress editing experience (or maybe I’ve just become used to it, coding and submitting PR’s on Github for the last 2 years).

I did like that Ghost was written in NodeJS, which I love far more than PHP, but in the end I did very little actual core hacking so that really didn’t matter.

 

Get docker container start time in unix timestamp (seconds)

You can use the following command to get the start time of a docker container in a unix timestamp (seconds since 1970).

docker inspect --format='{{.State.StartedAt}}' <CONTAINERID> | xargs date +%s -d

Just replace <CONTAINERID> with the actual container id.

Or you can use this to get the start time of all your containers:

docker ps -q | xargs docker inspect --format='{{.State.StartedAt}}' | xargs -n1 date +%s -d
 

How to stop Datadog alert recoveries going to Pagerduty

At Cloud9 we use Datadog for all our server monitoring and Pagerduty for alerts when things break. To do this we use the standard Datadog + Pagerduty integration and make Pagerduty automatically trigger for critical incidents by adding @pagerduty into the “Say what’s happening” field in the Datadog monitor.

Unfortunately datadog triggers the monitor both when it starts and when it has recovered. Because we had @pagerduty in the “Say what’s happening” area this meant we got a pagerduty call both times.

You can fix this by wrapping the @pagerduty trigger with {{#is_alert}}{{/is_alert}}. So your monitor should look something like:

Docker is having trouble creating containers. Please investigate @slack-datadog @slack-warnings {{#is_alert}}@pagerduty{{/is_alert}}

You can also use {{#is_warning}}@pagerduty{{/is_warning}} for warnings (where the monitor has gone over the warning threshold but not the alert threshold).

Then you can go back to bed safe in the knowledge your server isn’t going to wake you up to tell you “Everything is good, nothing is broken”.

 

How to get docker ps data in JSON format

If you want to get the output of docker ps in JSON format the easiest way is to run the remote docker API and use that, you can read about how to do that here

However if you can’t run the remote API because of security issues or for other reasons there is another way! With curl version 7.40 (If this isn’t available on your distro click here) and newer you can get data from the local unix socket, and docker always runs the remote api on docker.sock.

Here’s what you have to run:

curl --unix-socket /var/run/docker.sock http://localhost/containers/json

Tada! Nice JSON Formatted docker ps output. I recommend using the jq library if you want an easy way to parse it in bash.

 

Linux Traffic control hfsc what is [default $CLASSID]

While learning our traffic control setup at Cloud9 I came across this line:

tc qdisc add dev $IFACE root handle 1: hfsc default 12

Most of it is explained by these great in depth guides to tc and hfsc traffic shaping:

http://linux-tc-notes.sourceforge.net/tc/doc/sch_hfsc.txt
http://linux-ip.net/articles/hfsc.en/

And the man page at:

http://manpages.ubuntu.com/manpages/trusty/man8/tc-hfsc.8.html

The only thing I couldn’t figure out was what default 12 means. I originally thought it was some inbuilt default policy that comes included with tc. But it actually refers to the class that you’ve defined that it should fall back to using when nothing else matches.

So in our case we had the following other classes defined:

tc class add dev $IFACE parent 1:1 classid 1:10 hfsc rt m1 $HALF_MAXUP d 10ms m2 $MAX_UPRATE
tc class add dev $IFACE parent 1:1 classid 1:11 hfsc ls m2 $HALF_MAXUP ul m2 $MAX_UPRATE
tc class add dev $IFACE parent 1:1 classid 1:12 hfsc ls m2 $LOW_MAXUP ul m2 $LOW_MAXUP
tc class add dev $IFACE parent 1:1 classid 1:13 hfsc ls m2 $VERY_LOW_MAXUP ul m2 $VERY_LOW_MAXUP

$IFACE is the interface we’re running the rules on and the rates such as $HALF_MAXUP are network speeds in kbps that we’ve set and are still experimenting with.

Below this we then had a bunch of rules that prioritize different traffic types which look like below:

# prioritize SSH
$TC filter add dev $IFACE protocol ip parent 1: prio 1 u32 match ip sport 22 0xffff flowid 1:10
$TC filter add dev $IFACE protocol ip parent 1: prio 1 u32 match ip dport 22 0xffff flowid 1:10

# prioritize DNS
$TC filter add dev $IFACE protocol ip parent 1: prio 2 u32 match ip sport 53 0xffff match ip protocol 0x6 0xff flowid 1:10
$TC filter add dev $IFACE protocol ip parent 1: prio 2 u32 match ip dport 53 0xffff match ip protocol 0x6 0xff flowid 1:10

# prioritize application traffic
$TC filter add dev $IFACE protocol ip parent 1: prio 3 u32 match ip sport 8080 0xffff flowid 1:11
$TC filter add dev $IFACE protocol ip parent 1: prio 3 u32 match ip sport 8081 0xffff flowid 1:11
$TC filter add dev $IFACE protocol ip parent 1: prio 3 u32 match ip sport 8082 0xffff flowid 1:11

# Make UDP and ICMP really slow, they are rarely used for legitimate purposes
$TC filter add dev $IFACE protocol ip parent 1: prio 4 u32 match ip protocol 17 0xff flowid 1:13
$TC filter add dev $IFACE protocol ip parent 1: prio 4 u32 match ip protocol 1 0xff flowid 1:13

These rules ensure our users are able to ssh, resolve sites and host their applications at full speed, while throttling those who may be attempting to use Cloud9 for nefarious reasons. We also have other blocking rules not included here for security reasons.

So all outbound traffic that doesn’t match these rules is subject to the rules of class 1:12 which gives users $LOW_MAXUP bandwidth with a $LOW_MAXUP burst speed for uploads. That’s what the default 12 at the end means.

Let me know if you have any questions about this or suggestions on how to improve our traffic shaping. I’m no expert on this and didn’t write most of these rules, but just sharing what I’ve learnt to help others having the same confusions.

 

Using Express 4 routes to secure your web app

Today I had the fun task of taking Cloud9’s build bot and making it more secure. Primarily because it’s now exposed to the outside world and we don’t want random strangers having the ability to ship or revert our code.

Our bot responds to slash commands on Slack, so we can type /ship [appname] at any time in any channel in slack and the latest tested code will be pushed to production. It also recieves notifications from Jenkins when jobs have started, succeeded or failed.

Securing Slack

The first step was ensuring all Slack commands were actually coming from Slack. Whenever you create a new slash command Slack tells you it will send a specific token with all api calls, and you should use this to verify the call is from Slack.

Now there are multiple routes we wanted our bot to talk to and multiple slash commands to reach them, each with their own tokens. But we don’t want to add if (token == ‘xyz’) to every single route. Firstly because it’s messy, and secondly because then whenever a new developer joins the project they have to remember to do that or they’ll compromise security. So how do we do it? By creating a /slack route that verifies every token for us.

var express = require("express");
var config = require("config);

function verifyToken(req, res, next) {
if (!req.body.token || config.get("slack.tokens").indexOf(req.body.token) === -1) {
return next(new Error("Invalid slack token" + req.body.token));
}
next();
}

var slackRouter = new express.Router();
slackRouter.use(verifyToken);

slackRouter.post("/highfive", highFive.handleRequest.bind(highFive));
slackRouter.post("/ship", ship.startShipping.bind(ship));

app.use("/slack", slackRouter);

We have an array of possible slack tokens stored in our config file and whenever we add or remove commands we can simply add the token to that one list.

Now our routes are /slack/ship and /slack/highfive and whenever anyone sends data to them it will always validate that they have a valid slack token. No more manual verification in each route or having new developers forget to add security to their route, it’s all automatic.

Securing Jenkins

Our bot also listens to build hooks from Jenkins so that it can post to our Slack channel letting us know about the stats of various jobs.

We can secure Jenkins in the same way, but because it doesn’t pass any custom data in these job notifications we’ll secure it based on the requester IP address.

var requestIp = require("request-ip");

function verifyIPIsJenkins(req, res, next) {
var reqIp = requestIp.getClientIp(req);
if (!reqIp || config.get("jenkins.ips").indexOf(reqIp) === -1) {
return next(new Error("Jenkins push request came from " + reqIp + " which isn't a known address"));
}

return next();
};

var jenkinsRouter = new express.Router();
jenkinsRouter.use(verifyIPIsJenkins);

jenkinsRouter.post("/success", jenkins.buildSuccess.bind(jenkins));
jenkinsRouter.post("/failed", jenkins.buildFailed.bind(jenkins));

app.use("/jenkins", jenkinsRouter);

Now just like above we have 2 routes at /jenkins/success and /jenkins/failed and whenever anyone tries to access them it automatically verifies they are our CI server. If they are not the request will fail.

The reason I enjoy using these routes is they keep the code neat and also ensure that when someone comes to work on this project in the future they can easily add another route and won’t accidently allow hackers in the back door. Keeping things simple and automatic so any developer can pick up this code and run with it is my style of programming.

 

Above the fold is stupid

A short post for those in Internet Marketing who still believe the “Put everything above the fold because people don’t usually scroll down” myth. Unfortunately whoever originally said this didn’t quite think the data through and it has been repeated ad-nausium by Internet Marketers ever since.

Here’s why you shouldn’t worry about putting your stuff above the fold and it may even be counterproductive to jam it all up there:

  1. Most people aren’t interested in what you’re selling. Especially if your traffic isn’t pre-sold or coming from relevant locations. Not all traffic is equal and I really wish people would stop a/b testing like it is.
  2. 60% of people don’t scroll down because they don’t like what you have to sell, not because they don’t know how to scroll.
  3. Putting your signup form above the fold is counterproductive. Just because people don’t scroll dosn’t mean they’re going to be like “Oh wow a signup form, I better add my details”.
  4. If you had better content or marketing in place of that signup form you’ll probably (test this!) get better results as you hook people in and get them to register later.

The core thing you should be putting above the fold is content that sucks your visitors in, something that within 5 seconds makes them think “Hey this is a problem I have and they may have the solution”.

Here are things that should not be above the fold:

  1. Videos that start slowly (boring)
  2. Signup forms (seriously the only person who would sign up without reading is your Mum, or heavily presold visitors, in which case it doesn’t matter where your signup form is).
  3. Pictures that mean nothing to fresh visitors (your logo, stock photography, banners etc).
  4. Links (If you want your visitors to do something don’t overload them with options).

Now go build a landing page people actually enjoy and remember: test everything. If you think I’m wrong run a test then come back here and tell me how wrong I am. Being told I’m wrong is fantastic, because it helps me constantly grow and be a better market and human being.