Using D3 for realtime webserver stats

I wanted to learn a bit of D3, so I made a small web app that shows (nearly) realtime webserver statistics as pretty graphs. Here’s a demo for my server and a setup guide.

Source on Github Source on Github
(You need both, but the first will get the second via NPM.)

Client code

MediaTemple were nice enough to introduce an API that provides a load of stats about what your server is doing some time ago. This is the same info that you can see in the server status bit of the account centre. Unfortunately the interface there is a bit clunky and it also logs you out (sensibly) after a few minutes’ inactivity.

Inspired by some of the really cool stuff Mike Bostock has done with D3, (mt) Stats gives you the same info with a (hopefully) nicer interface, and maybe even doing quite well with Tufte’s data-ink ratio. You can add as many graphs as you like (if your CPU can take it at least), move them around by dragging and adjust the interval over which stats are shown with the range slider.

If you hover over a graph, you should see buttons to change the metric for the graph and use a log or zero-based scale. On my server at least, I often end up with some large peaks in the kb out graph, which can hide a lot of much smaller bumps when only a few kb was transferred, so a log scale can be a useful way to see them more clearly. Similarly, certain metrics like memory usage can work better with (or without) a zero-based scale.

The app settings are stored in your browser’s localStorage (under mt-stats), so if you close the tab they should be there when you revisit the site. Though obviously not if you use a different browser or clear the cache.

I used Backbone to look after (and modularise) the graphs collection and some jQuery UI goodness for the drag and drop and slider. The site uses Twitter’s Bootstrap, lightly restyled to give it a Metro-like appearance. (Much easier than last time as the look has now been rolled out beyond phones.)

Server code

The back end is written in Node. The reason there’s a server component at all is to hide the API key away from prying eyes, though I did end up abstracting away some of the API’s quirks too. (The same API key can be used to do all sorts of other things like reboot the server and change the root password. I wish it were possible to create keys at different privilege levels but it’s not.)

The stats API can get details of the server’s current state or its state for an interval in the past, in which case a resolution can be given. This resolution may or may not be respected by MediaTemple, e.g. there’s a minimum resolution of 15s and a request for 2 hour intervals will be returned at a resolution of 60 minutes.

The API server will also only serve up a fairly small amount of data at a given resolution – smaller than I’d like, so the server divides up the client’s range into several requests and combines them before returning. The server code’s currently set up to return up to a week’s worth of data at once; it will return more, but at a resolution that would hammer the MediaTemple server rather, so please add some more ranges to the mt-stats.json config file if you want to do that. All data are transferred using JSON.

Setting up (mt) Stats on your server

  • Get an API key for your MediaTemple server. You’ll also need your service ID later, which you can get by visiting https://api.mediatemple.net/api/v1/services/ids.json?apikey=XXXXX (I only have one service ID as I have one server, but apparently you could see more.)
  • Get a copy of Node.js. You can put (mt) Stats on your server if you want to access your stats from anywhere, or run it locally on your machine if you prefer. If you haven’t got Node already and you’re using Linux, there’s an installation guide here. If you’re running Windows or OSX then you can just download the installer from the Node site.
  • Download (mt) Stats Viewer from Github and run npm install in the folder where you put it to get the packages it needs (list further down for reference).
  • You should now have a copy of mt-stats. Rename the config file node_modules/mt-stats/config/mt-stats-sample.json to mt-stats.json and change the service ID and API key to match your server and key.
  • Rename the config file config/app-sample.json to app.json and change the URL and port. localPort is the internal port on which the service runs and url is the external URL for the service. Also edit gaAccount and gaDomain to match your Google Analytics details if you want to track usage.
  • Run app.js and you should be good to go.

mt-stats-viewer settings

Please rename app-sample.json to app.json.

  • localPort – Local port on which the app is running
  • url – Public URL for the app
  • modedevelopment uses uncompressed and (many) separate JavaScript files, while production uses Google’s CDN for jQuery
  • title – Page title
  • description – Meta description contents
  • graphRanges – Timespans that are sent to the client for the range slider. Currently 5 mins – 1 week in ms
  • author – Meta author contents
  • jQueryCdnUrl – Yup
  • gaAccount – Your Google Analytics account
  • gaDomain – Your Google Analytics domain
{
	"localPort": 3000,
	"url": "http://bits.meloncholy.com/mt-stats",
	"mode": "production",
	"title": "(mt) Stats",
	"description": "Visualise your MediaTemple stats with some gorgeous D3 graphs.",
	"graphRanges": [
		300000,
		600000,
		900000,
		1200000,
		1500000,
		1800000,
		2700000,
		3600000,
		5400000,
		7200000,
		10800000,
		14400000,
		18000000,
		21600000,
		32400000,
		43200000,
		54000000,
		64800000,
		75600000,
		86400000,
		172800000,
		259200000,
		345600000,
		432000000,
		518400000,
		604800000
	],
	"author": "Andrew Weeks",
	"jQueryCdnUrl": "http://ajax.googleapis.com/ajax/libs/jquery/1.8.0/jquery.min.js",
	"gaAccount": "UA-XXXXXXXX-X",
	"gaDomain": "meloncholy.com"
}

mt-stats settings

Please rename mt-stats-sample.json to mt-stats.json.

  • serviceId – Your server’s service ID
  • apiKey – Your MediaTemple API key
  • mode – Doesn’t do anything here, but its presence gives me a warm, comforting glow
  • rootPath – The root URL path for all API calls
  • interval – Server polling interval for client. MediaTemple’s stats update every 15s
  • ranges – For each range, the resolution at which to request data from the API (e.g. every 15 seconds) and the maximum span (step) to request in one go (to stop the API server objecting). Range is the maximum timespan at which to use that resolution and step
  • metrics – Metrics supplied by the API. apiKey is the key name in JSON objects and niceName is the name to use on the graphs
  • definedRanges – MediaTemple also supports some default intervals that can be requested with these URLs, e.g. this URL will serve up the last 5 minutes’ data. These are not used by the front end
  • currentUrl – API server URL from which to get the current stats. %SERVICEID and %APIKEY are replaced with your service ID and API key
  • historyUrl – API server URL to request stats going back for the past X seconds, e.g. this URL will also give the past 5 minutes’ data
  • rangeUrl – API server URL to get stats covering a specified time range
{
	"serviceId": 000000,
	"apiKey": "XXXXXXXX",
	"mode": "production",
	"rootPath": "/api/",
	"interval": 15000,
	"ranges": [
		{ "range": 3600, "resolution": 15, "step": 3600 },
		{ "range": 43200, "resolution": 120, "step": 28800 },
		{ "range": 86400, "resolution": 240, "step": 57600 },
		{ "range": 604800, "resolution": 1800, "step": 432000 }
	],
	"metrics": [
		{ "apiKey": "cpu", "niceName": "CPU %" },
		{ "apiKey": "memory", "niceName": "Memory %" },
		{ "apiKey": "load1Min", "niceName": "Load 1 min" },
		{ "apiKey": "load5Min", "niceName": "Load 5 min" },
		{ "apiKey": "load15Min", "niceName": "Load 15 min" },
		{ "apiKey": "processes", "niceName": "Processes" },
		{ "apiKey": "diskSpace", "niceName": "Disk space" },
		{ "apiKey": "kbytesIn", "niceName": "kb in / sec" },
		{ "apiKey": "kbytesOut", "niceName": "kb out / sec" },
		{ "apiKey": "packetsIn", "niceName": "Packets in / sec" },
		{ "apiKey": "packetsOut", "niceName": "Packets out / sec" }
	],
	"definedRanges": ["5min", "15min", "30min", "1hour", "1day", "1week", "1month", "3month", "1year"],
	"currentUrl": "https://api.mediatemple.net/api/v1/stats/%SERVICEID.json?apikey=%APIKEY",
	"historyUrl": "https://api.mediatemple.net/api/v1/stats/%SERVICEID/%RANGE.json?apikey=%APIKEY",
	"rangeUrl": "https://api.mediatemple.net/api/v1/stats/%SERVICEID.json?start=%START&end=%END&&resolution=%RESOLUTION&apikey=%APIKEY"
}

Dependencies

If NPM doesn’t do it for you, you’ll need to get copies of

Problems and stuff

It doesn’t work at all! I’ve pressed new and nothing happens!

IE and Opera aren’t supported (at least up to IE 9) – as I said, I wrote this principally to learn about D3 and for me, so supporting older browsers was not on my to do list. The biggest problem here is probably D3, which uses the browser’s SVG capabilities to draw graphs and doesn’t work with IE and Opera. (I know that IE (as of version 9) and Opera have some SVG support, but D3 doesn’t seem to like it, or at least not for the graphs I used. I haven’t looked into this further.) If this bothers you, I’d be more than happy to integrate a polyfill if you write one. :)

Why do the graphs jump sometimes?

The client requests an update every 15 seconds (the maximum resolution offered by MediaTemple) and it relies on this to slide the graphs along and create a smooth viewing experience. Unfortunately the updates from MediaTemple are not always that reliable: they can have a timestamp that’s off by a couple of seconds or, sometimes, by 30+ seconds. In these cases, the graphs will unfortunately jump as they recalibrate to the new timestamp.

What’s with the ugly loops in my graphs?

Particularly after loading the page, MediaTemple’s API will sometimes supply an update that’s timestamped 30-45s after the previous one, rather than the 15s the app was expecting. The line tension I’m using means that the graphs sometimes show little loops in these cases. If this bugs you, you can adjust it in mt-stats.graph.js.

Urg! My graphs just raced of the screen and vanished!

The graphs sometimes seem to get their knickers in a twist and fly almost entirely off the screen before new data points are added. I think this is because the API server sometimes returns very few data points (2 or 3) when the range is set to 5 mins, though I have seen it happen at other times too. But I’m not sure what’s going on here. Sorry.

The axes are weird / have disappeared

Yes, they seem to do that sometimes. This can happen if you’re using a log scale and the graph range is currently small and not zero-based (e.g. 88.5-88.6). In those circumstances, D3 finds it difficult to add suitable tick marks. But it happens at other times too, and in those cases I’m not sure if it’s my fault (probably) or a bug in D3. Refreshing the page could help, but if not I apologise.

I don’t want to run Node.js

I like Node so I used it here. That said, the back end is basically a specialist proxy server, and translating it into another language should be pretty easy if you want to use PHP, Ruby or something else. (Again, please let me know!)

It only works on MediaTemple

That’s true. But if you use another host that provides server stats in a JSON wrapper then translating it should be fairly trivial. MediaTemple sends status updates in these formats:

Ranged request

{"statsList":{"timeStamp":1343138160,"resolution":15,"serviceId":00000,"stats":[
{"timeStamp":1343138160,"cpu":12.99,"memory":83.24,"load1Min":0.0,"load5Min":0.0,"load15Min":0.0,"processes":48,"diskSpace":43.88,"kbytesIn":1.93,"kbytesOut":21.07,"packetsIn":11.4,"packetsOut":20.27,"state":1},
{"timeStamp":1343138175,"cpu":0.01,"memory":83.79,"load1Min":0.0,"load5Min":0.0,"load15Min":0.0,"processes":48,"diskSpace":43.88,"kbytesIn":0.85,"kbytesOut":33.61,"packetsIn":16.07,"packetsOut":22.6,"state":1},
...
]}}

Current status request

{"stats":{"timeStamp":1345595595,"cpu":1.76,"memory":103.39,"load1Min":0.0,"load5Min":0.02,"load15Min":0.0,"processes":72,"diskSpace":49.2,"kbytesIn":57.0,"kbytesOut":6.2,"packetsIn":72.0,"packetsOut":86.0,"state":1}}

I think the only thing you’d have to do to the front end is change this line in mt-stats.data.js to extract the stats from whatever object your server sends them in

for (var i = 0, len = data.statsList.stats.length; i < len; i++) {

You'll have to do a bit more fiddling around in mt-stats/index.js and mt-stats/config/mt-stats.json as that uses MediaTemple URLs and checks for specific return messages, but hopefully translating wouldn't be too much work.

En fin

Hope this is useful / interesting to someone other than me. If you end up using it, please let me know!

Source on Github Source on Github
(You need both, but the first will get the second via NPM.)

Legal fun

Copyright © 2012 Andrew Weeks http://meloncholy.com

(mt) Stats is licensed under the MIT licence.

Let me know what you think

Your email address will not be published. Required fields are marked *