This is Part 1 in the series, Automation.
Summary: After a comical disaster with Zapier, I contemplate creating my own bot to manage my publishing workflow
For a couple of months, I’ve been using Zapier to cross-post content from my blog to my Twitter and Facebook accounts. I only use the free tier, thinking I’d never reach the 100 monthly task limit given that at most I publish a few articles a week.
So far, so good. That is until I decided it was time to reorganise my blog and introduce clean, date-based URLs.
The way the Zapier zaps I use work, is to read my RSS feed and create link posts when a new item is introduced. Common sense dictates that it should read the date field. Only if the date field changes should it kick of the cross-link.
Unfortunately, bots are dumb and instead Zapier interpreted the change to the URL field as a worthy excuse to fire off a barrage of Tweets and Facebook posts. Since I have over a hundred articles on my site, within a minute of rebuilding the site, I exhausted my monthly task quota.
My quota’s due to reset in three weeks.
Now, I can live without automatically cross-posting for a few weeks. I can substitute it temporarily for a simple Workflow on my iPhone. The problem is I have another Zap who’s job it is to catch specially formatted emails I send to my Gmail account and turn them into draft blog posts in Dropbox. It’s become an important part of my publishing process. That’s now dead too.
So, I’m left with a couple of remedies. I can wait three weeks and stew on my failings. I can cough up $20USD per month to Zapier to upgrade my quota and unlock more features (sorry, not happening). I can switch to IFTTT, which is free but somewhat limited.
Or, I can make my own bot.
Wait, what? Didn’t I promise earlier this year not to reinvent wheels?
Yes, I did, but in my own defence that was before I really upped my blogging output. The spirit of the post was to concentrate more on writing than on software development or getting hung up on process.
For the most part, automation has allowed me to do just that and it’s one of the reasons I’ve been able to write more, especially on my blog. I simply write and my minions, a mix of scripts and bots, take care of the boring crap.
At the moment though, the system has effectively broken. Not only has Zapier clipped my wings, but my build machine, a 2011 Mac mini, is on its last legs.
I can’t afford for my production system to be wonky. It’s reliant on third-party services and an ailing computer that can’t be trusted to do what I want when I want. Something needs to change.
So, a DIY production bot; a little minion to consolidate these disparate tasks and services that I can shove in a reliable server. Such a bot, would give me much more control over my production process and perform tasks that these service cannot do, or charge a premium for the privilege.
My needs are relatively few at this point and I’d start with replicating the Zaps I use now. My plan is to use Python, which is very well suited to creating bots that interact with web service APIs.
The web services I need to talk to are:
Python has a wealth of third-party modules and all these web services are well supported. Hacking together a basic bot will take me half a day at most to get something up an running.
The knotty problem is working out where it would run. I can either use a physical server, or a virtual one up on DigitalOcean.
Digital Ocean is an attractive option because it’s cheap, reliable and I already use it. My only concern is that last time I ran the Dropbox client on a $5 droplet it kept running out of memory and crashing. Maybe I can jus use the API instead of the full client.
Physical hardware hosted inside my home LAN will give me more resources (RAM, CPU, storage) but will be at the mercy of my dodgy ADSL connection. There’s also the problem of what hardware I use. If I’m using a box 24/7 not only does it have to be reliable, it also has to be cheap to run.
My first thought is a Raspberry Pi 2, which I have lying around after it killed the last SD card I gave it. It’s certainly cheap to run but I’m not confident of using it in production unless I move the file-system off the SD card. The Raspberry Pi, being an ARM-based computer, also doesn’t suppor the Dropbox Linux client, meaning I’d be relying on the API.
Another possibility is buying either an Intel Nuc, or perhaps an Intel Compute Stick. They are quite energy efficient, run Linux and being Intel x86-based they can also run the Dropbox client without issue.
Anyway, it’s given me food for thought. There’s a few more ideas (pretty vague at the moment) that I’m toying with. I’ll park this for the time being; I’ll spend this weekend writing instead.