MonthSeptember 2014

DevLog Sep 12-24, 2014

That was a long break. Should get into better habits. I’m sure I’d miss some here. Oh well.

It was a very interesting week, culminating in some very interesting things at Trafalgar Square. I feel much better as a person and more calmer/chilled out now :) Should remember to take weekends off.

  • Helped the WikiMetrics people with some puppet changes. They were requiring mysql directly instead of using the class we have in ops/puppet, which meant it put data in the /var partition instead of in /srv. This was because they want to be a ‘proper’ independent puppet module, not depending on the other modules in ops/puppet (since the module is used in vagrant as well). Puppet has terrible support for this kind of thing, and ops/puppet by itself also suffers from an incredible amount of NIH syndrome when it comes to modules. Some of the NIH-so-rewrite is justified, but I don’t think all of it is. Oh well. Hopefully something that can be fixed over time…
  • Added basic icinga/graphite based monitoring for contint project (runs our jenkins installation). Quite trivial, but should be ported over to shinken at some point soon.
  • Contributed a fair bit to the Shout Web IRC project. I was looking around for an open source replacement to IRCCloud, and found a bunch of OSS projects (Subway and being the others). Shout seemed the most active (and simplest) of them all – I couldn’t actually get the others to a working installation. I fixed a bunch of code formatting issues, and also added a couple of features (Desktop notifications being the most significant, I think). The maintainer was fun to work with, and merged most of my changes quite promptly. I hang out on #shout-irc channel now, and activity seems to be speeding along. Should be fun to see how it goes :)
  • Moved off IRCCloud. Stopped paying for it. I’ve also stopped doing email on my phone, in an effort to be more minimalistic / less distracted all the time. This meant no more IRC on phone, so my primary reason for using IRCCloud disappeared. Am back to using LimeChat with my ZNC bouncer now. And nothing on my phone. In case of emergencies, I still have the IRCCloud app on my phone, and can use the free edition to connect to my bouncer temporarily.
  • Started working on labs-specific monitoring, with shinken – test instance at Discussion in the ops list about how to best implement this. Shinken is supposed to horizontally scale much better than icinga since it is implemented as a bunch of different daemons that can talk over the network. Right now they are just one machine, but will probably expand at some point not too far away. They are running as a labs project rather than on a physical machine (labmon1001) because it is much easier from a networking perspective. Right now it only has a simple http check for betacluster, but more coming.
  • Started refactoring all our icinga code. Separating them out into nagios_commmon module for config that can be shared between shinken and icinga, and into an icinga module for everything else. Moved all the custom checks and check config we have, moving other things as we go. This is a bit hard / frustrating since I don’t have access to the machine icinga runs on (neon), and also because I still do not have +2 in our ops repo (should change in a month, tho!). Am about 40 patches in (small ones!), let’s see how many more it takes!
  • Moved our dsh code into a module. Was fairly trivial. Now to find ways to get it merged :)
  • Started trying out pinboard. Interesting, barebones design. The bookmarklet is a bit slow to load, though. Let’s see how often I use this. (I did pay for it)

Five more days left in the UK :( Let me see if I can continue doing these when here, in a more prompt manner.

DevLog for Sep 10-11 2014

Back from an awesome vacation. Too awesome to write about, even :) Suffice it to say, England has some really pretty places.

Some Android app work, and lots of monitoring work

  • Fixed bugs causing the Wikipedia Android Alpha from building properly. Now it builds properly whenever there is a new commit. Hooray! This was primarily caused by me forgetting to give it lots of RAM (8G VMEM) to execute the mvn build commands ( and also not cleaning up previous .alpha subfolders ( – this causes a chain of .alpha.alpha.alpha.* subfolders, breaking the build.
  • Added a patch to the Android alpha app itself that checks for updates every day or so and notifies you if there’s a new one. Was fairly trivial to write, although I was hoping to make it more seamless (i.e. download the apk myself and just pop it up for people to tap). It now requires 4 clicks to install it, should be able to bring it down to 2 at some point in the future if people care enough.
  • Added a method to our check_graphite code that lets you individually check a bunch of metrics for thresholds ( This makes it much simpler to do icinga checks on a bunch of metrics that are all measuring the same thing but from different machines. BetaLabs and ToolLabs checks use this.
  • Cleaned up a bunch of minor things with our check_graphite script. Also fucked up trying to replace all double quotes in it with single quotes for consistency – it replaced the double quotes being used inside single quotes, and caused all checks to fail. Fixed shortly by
  • Added more monitoring for betalabs! Now checks for stale puppet runs ( and low space on the root partition ( All are green now, thanks to some work from bd808.
  • Added monitoring for ToolLabs! Now checks for stale puppet runs, low space on root and /var, and puppet failure events ( Also checks for high sustained CPU usage ( Then spent some time (with help from scfc_de (whose nick I kept spelling as scfe_de until today)) cleaning up the puppet failures. They are all green now as well.
  • Did a bunch of cleaning up work around the graphite role, removing the realm branching ( Ori says everytime realm branching code is removed, an angel gets its wings, so well done there.

Not a bad day, eh? I’ve been trying to wake up early, perhaps that is helping.

Shitty Web advertising #1


The message icon was flashing obnoxiously. I wonder how much money CNet made off that.

I don’t notice these on desktop thanks to Adblock. Should set it up on my phone too.

DevLog for 31 Aug, 1-4 Sep 2014

Missed DevLogging for a while.

Am in London now.

  • Started using a spare Majestouch Ninja 2 over my regular Kinesis Advantage. This is way more portable, and my hand does not seem to be hurting while using it (so far only about 4-5 hours). If this keeps up, I should be able to move to a similar smaller keyboard over the much bulkier kinesis. There’s still a little bit of discomfort, so I’ll probably want a very portable and mechanical split keyboard. Can’t sadly seem to find any, though :( Maybe I should just build one with an arduino :)
  • Setup icinga checks for puppet failures and disk space issues on betalabs, and fixed a bunch of issues/docs in our icinga puppet code during that time. This still doesn’t properly work since our implementation of check_graphite does not support wildcard metrics properly – it should check thresholds for each series, but it seems to do that only across the entire series combined, which is kinda useless. Should fix that soon by adding more features to it. Also might try out other alternatives to icinga, since our icinga puppet code is a fuckball anyway.
  • Fix a couple more Quarry bugs. There’s still a random bug where celery seems to be attempting to read data about a query run from mysql before the web has committed it, which is theoretically impossible (I do a commit before sending the task to celery with the id), so I suspect some mysql fuckery. Will need to debug that sooner than later, and also consider moving to postgres. But then Quarry will have to deal with SQLite (for result storage), MySQL (for connecting to labsdb) and with postgres for local data, which sounds insanely complex. I also added CORS support to resultsets, and Magnus is playing with it (wooohhooooo!!!). I’m going to add more features to make it easier for people to use results from quarry in their JS applications elsewhere. Should be fun.
  • Finished videos for the first week of the Coursera Data Analysis and Statistical Inference class I’m taking. Started poking around R since the labs for that class are from R, should be fun.
  • Chad has started his devlog. He does search stuff at Wikimedia and is a co-whiner about all things Java. Do check out :)

Am away on ‘vacation’ till Wednesday, yay! :) Should disconnect well.

DevLog for 29, 30 Aug 2014

Chill weekend. Didn’t really do anything code related. Recovering from friday night party :)

Started reading Data + Design which seems quite nice. Also starting a coursera course on Data analysis and Statistical Inference on Sep 1, should be fun.

© 2017 Yuvi Panda

Theme by Anders NorenUp ↑