Tips and Tricks for CircleCI

I am not a developer, but I work with developer tools all day. This can be frustrating, because a lot of tools made for developers assuming knowledge that I’m lacking.

Of course, if that wasn’t the case there would be less need for technical writers and my work would be harder to come by!

In any case, After spending the last few days migrating my job’s documentation to use the CircleCI 2.0 configuration, here are some tips I picked up:

Non-interactive shells

This can be tricky if you don’t keep it in mind. Because CircleCI tests are run in a non-interactive environment, your output can get …funky. For example, here’s the output of sculpin generate, which we use to turn our markdown files into html, running locally through CircleCI CLI:

Here’s the output from the build on CircleCI’s platform:

Sculpin assumes an interactive shell in which it can rewrite its output as it goes. CircleCI’s environment is set for better logging, not better human readbility (as it should be).

This is of course a trivial example, as I could add --quiet to my generate command to make my CI output nicer. But that’s not always the case…

Rsync output on CircleCI

Part of my build process involves copying my compiled docs to a staging environment. Our scripts use rsync, the most common choice for transferring files from A to B.

The Problem

No matter what permutation of logging flags I gave rsync, the only output I could get from my CI builds was empty, or a list of every single file it was checking for changes:

Not a critical problem, but it certainly reduces the usefulness of the output.

The Solution

While I could pipe the rsync command directly through grep, that would interfere with the while loop I put rsync in, which checks for exit codes to determine success. Instead, I constructed my rsync command as follows:

rsync --checksum --delete-after -rtlzq --ipv4 --info=BACKUP,DEL --log-file=multidev-log.txt -e 'ssh -p 2222 -oStrictHostKeyChecking=no' output_prod/docs/ --temp-dir=../../tmp/ $normalize_branch.$STATIC_DOCS_UUID@appserver.$normalize_branch.$STATIC_DOCS_UUID.drush.in:files/docs/

That’s a lot to unpack, I know. The key flags are -q, which keeps the output quiet, and --log-file=./multidev-log.txt, to redirect output into a log file. Then, once my loop has exited, I can run:

cat ./multidev-log.txt | egrep '<|>|deleting' || true

My pipe to egrep searches for the characters < or >, or the word deleting. Using the OR operator || true means that even if the grep returns no results the line will still exit with code 0, keeping my script from failing.

Now my output is concise and useful!

(By the way, I learned a lot of useful information about rsync output from this blog post, run through Google Translate.)

Dockerize for background processes

IMHO, this tool is poorly named. While it’s purpose was originally for CI builds involving multiple docker containers (I assume), it’s helpful even in a single container environment.

The Problem

I was seeing intermittent build failures coming from my Behat tests:

Behat is configured to look at port 8000 for my Sculpin server to serve the files it tests. The Sculpin step wasn’t failing, but rather it wasn’t starting up fast enough.

The Solution

My initial response to this problem was to add sleep 5 to my Behat step, giving Sculpin more time to initialize. But then my friend Ricardo told me about Dockerize. Because my Docker image is built from a CircleCI image, it was already installed. So I added a new step in .circleci/config.yml:

- run:
    name: Start Sculpin
    command: /documentation/bin/sculpin server
    background: true
- run:
    name: Wait for Sculpin
    command: dockerize -wait tcp://localhost:8000 -timeout 1m
- run:
    name: Behat
    command: |
    /documentation/bin/behat

Dockerize checks for a service listening on the port, and retries if one isn’t found:

Now I have one less reason for my builds to fail…

 

Do you have any tips, tricks, or answers to common “Gotchas” for continuous integration testing? If so, please share them!

Make a site, NOW!

A month or two ago I registered eastcoastamphicar.com and set up a WordPress site on Pantheon. The eventual goal was to make a web presence for my friend Billy, who restores the historic amphibious cars, and gives rides to folks whenever he can.

Today he called me to let me know that he’d be filming tomorrow for CBS new out of New York. He said he’s doing it for the exposure, that they’d plug his email and phone number at the end of the segment.

Wait, what? His email is at yahoo.com, and his phone is a land line, no texts. Sounds like he needs that website, and he needs it now!

So I took some drone footage a friend captured, some photos I’d scanned from Billy, and made an MVP for eastcoastamphicar.com. And hopefully in another hour, once Namecheap renews their DNS records, I’ll have a nice professional email address for Billy to give CBS.

Oh, look at that. It’s past midnight. Off to bed!

Idle Hands are The Hipster’s Plaything

I’m a pretty modern person. I work remotely for a tech company, I’ve lost count of the number of computers in my home (laptops, old towers, Raspberry Pi’s, etc), I even have Tux tattooed on my arm.

And yet I find myself falling into the hipster trappings of my generation; I have a turntable, I vape, and I just bought a french press. And I wonder to myself, why am I doing this? I’m a scarf and v-neck shirt away from hipster bingo!
Continue reading Idle Hands are The Hipster’s Plaything

Organizing Photos

This post is on quickly sorting photo libraries. If you want to skip the backstory and jump right to the commands, click here.

I recently switched my personal file hosting/sharing to NextCloud,  from OwnCloud. Among other features and improvements, it will automatically sort uploaded photos from my phone into folders by date. 

This great new feature created a glaring contrast with the many years’ worth of previous photos, mostly all in a single directory, some sorted manually by event.  The process of grammatically identifying, sorting, and deduplicating was simple enough once all the tools were lined up, but finding them was not. What follows are the tools I used to get my photos in order.

Continue reading Organizing Photos

How to ask for help on IRC

Hi.
Hello?
Anyone there?
My server broke, can anyone help?

As a regular IRC user, I’ve seen this message many times. I’ve also seen the hateful responses these intrusions generate. IRC is an older communication form (relative to the Internet), mostly inhabited by those who’ve used it for decades, and are set in their ways.

IRC is also where you’ll find large communities of people all interested in specific topics, usually tech-related. If you’re an expert on MySQL, Ubuntu Linux, Python, etc, you’re most likely found in #TOPIC on at least one IRC network.

Which is why there are newcomers everyday, logging on to seek the wisdom of these software gurus. If you’re one of them, this post is for you; the IRC amateur who’s venturing to Freenode or OFTC for the first time to ask for some much-needed help.
Continue reading How to ask for help on IRC

Using Nest’s API – Part 1

I’ve been made an interesting offer by Nest – they will give me one of their spectacular thermostats (and a smoke / CO detector), if I can find a fun open-source use of their API and write it up.

Challenge Accepted

I already explained that I’m not a programmer, although I’ve been experimenting with python for the last couple of weeks, but I’ll happily give it a shot.

Step 1: Brainstorming.

What would I want to build to extend the functionality of a wifi-connected smart thermostat? The first thing that comes to mind is making a twitter account for it, and having it tweet the temperature in my home. I figure’d that would have been done so many times before that there would be nothing left to create, but a quick google search told me that was not the case.

What else? Well, I have a promo code for Twillio, and Nest’s API has functions for smoke and CO detection. I see an opportunity to create an alert to my phone if there’s an emergency at home. If the detector is downstairs, I’d rather it call the phone on my nightstand then sound an alarm where it is.

Do you, the avid reader, have any suggestions? What would you want your thermostat to do for you?

 

 

Penapps XII

From now through Sunday I’ll be at Pennapps with Linode, giving out swag, Linode credit, and mentoring where I can! I may also try to do some hacking myself, but I’m not sure if I should be reorganizing my services across different servers, or working on a new project.

These hackathons run 48 hours straight, but I don’t know how long I’ll last. I’ll try to update here as the hours pass.

EDIT: This is what I’ve come up with so far: Python Prime Number Generator

EDIT #2: I’ve made another couple of useless site: GET SCHWIFTY! and MR MEESEEKS

Lazarus’d

Hello, avid reader! (Don’t question the adjective, let a man dream). The good news is that I’m back up and running with a site that’s more than just HTML edited in nano. The bad news is that my older posts are probably not coming back. I may be able to recover the text from a backup somewhere and manually re-import them, but no one should hold their breath over it.

As I type this, my coworker and I are entering the dreary early morning hours of a hackathon in Brooklyn, where we are stationed as mentors. My personal hacking has been the recreation of this blog, and installing CyanogenMod on my phone.

The Red Bull is calling out to me, but with the Starbucks and Monster already coursing through my veins, I’m saving it until I start to nod off, in the hopes that I’ll make it through to the judging and still be sane enough to choose a worthy winner for our sponsor prize.

The following hours will probably be filled with further CMS and ROM tweaks, while fighting the sweet embrace of sleep that seeks to pull me away from my duties and into the warmth of nothingness. Toodles!