Warning: the below post is probably long, and definitely geeky. You’ve been warned.
I’ve had this blog and several others hosted on a server I run and pay for since around the neighbourhood of 2010 or 2011. Naturally, this means I go beyond the whole finding random things to post about idea and dip into the territory of the sysadmin. Awesome, insofar as experience goes–not, mind, that said experience gets me any closer to being employed, but you’ll have that. But the more I play around with it, the more I think it gets me ready/comfortable with the idea of actually doing something like this and getting paid for it. Besides, I like a challenge.
So I’ve been running this particular server since August of 2012, or thereabouts. And in that time, yeah there’s been just a tiny little bit of broken here and there. But I usually had some warning or could guess that, hey, what I’m about to do will very likely end in spectacular fashion with me spending the next week and a half picking the pieces off my floor. This time, not so very much.
I maintain a small platform where I can stick random bits of info, like documentation for things I’ve figured out about otherwise less than stellarly documented programs. Or, you know, random things that just might turn out to be useful to me a year and a half later. That platform is powered by MediaWiki, who’s probably best known for being the thing used by Wikipedia. So you know, it’s been poked at, prodded at, tested the hell out of, and generally considered stable enough. Well, that or Wikipedia is partially owned by MediaWiki, but hey whichever. So I figure, why not? It’s scaleable, so my small little documentation platform oughta be no sweat. Which is largely true, until it breaks.
I’d never actually bothered digging into the code, if I’m being honest. I figure eventually I’ll get to it, then things happen, and it doesn’t really get gotten to. You know the deal. Fixing the broken, though, necessitated a quick little dig through the surface layer of code. The bright side: now I know why it’s relatively light on database usage. Can I trade, now?
Here’s a little bit of a primer, if you’re one of those folks who’re on the border of techy but not quite ready to slide across it yet. Most software, like wordPress for instance, pretty much leans on whatever database you’ve set up for it. Everything hits the database, no questions asked. Unless you run some kind of a caching plugin (I do), even the basic trying to access the website hits the database. Database goes down, site goes down. MediaWiki does that, to a point, but there are enough layers between the database and you that it’s not entirely obvious. One of those layers is the extensive use of regular expressions for damn near everything. Almost nothing in the software is actually pulled from the database after, perhaps, the first initial load. Exceptions might be made for things like menus, but that might also be stored in the code itself somewhere and I just haven’t bothered finding it yet. But everything else, like for instance the actual page content? Cached somewhere on disk, then hit with a regular expression. Awesome, in theory. Works perfectly, again also in theory. Until theory goes out the window and they release a server software update that pretty much breaks the place. I applied that server update. Had no idea anything was broken–because barely anyone uses what I’ve set MediaWiki up for, and nothing else went sideways. So all was right in the world. Until my documentation actually needed to be flexed.
In fixing the broken, I learned exactly two things, real quick. Thing the first: Even on non-Windows systems, updates still break pretty–I knew that already, but it’s occasionally nice to have that confirmed once in a while. Especially when you know a few people who’ll gladly insist they’ve never had an update problem with $OtherSystem like they’ve always had with Windows. And thing the second: If you release an update to a pretty significant piece of software that breaks compatibility in new, interesting and creative ways, and pretty much no one sees it coming, you’re doing it wrong.
Let the record reflect I still love the sysadmin gig. Let the record also reflect I’d still love to be paid for the sysadmin gig. But I’m kind of wondering now how many paid sysadmins are sitting in an office wishing they could fire themselves a developer. Other people’s broken is never a fun thing to come home to. Now, I speak from experience.