You’ve seen the Medium posts, you’ve seen the rosy tech-talks. And you know that the next big thing in your mobile testing toolkit needs to be screenshot tests. But you quickly hit roadblocks getting your team to use it. In this guide, we’ll help you understand and navigate these roadblocks.
But before that, let’s address screenshot vs snapshot tests. Most Medium posts will tell you that snapshot tests are more than just screenshot tests… well, they’re partly right and mostly wrong. In reality, the iOS world calls it snapshot tests, and the Android world calls it screenshot tests. You see, back in 2015 when I built and open-sourced screenshot-tests-for-android, the iOS world already had ios-snapshot-test-case, built by my colleagues at Facebook. I was too lazy to rename the main Screenshot class, and so I decided to call it screenshot testing instead of snapshot testing. screenshot-tests-for-android was the first screenshot testing library on Android, and the name stuck ever since on Android across multiple libraries by different people.
But I digress, let’s convince your team to use screenshot testing.
Your immediate team
Alright, so you’re excited after reading that fantastic Medium post, and you share it on your Slack: “We need this now!” Your manager is like, “Oh great, let’s do it”, and you’re like … what now?
Maybe you write your first screenshot test for your product screen, and commit the test and the screenshot to the repo. Hi-fives all around. Then Ben over there tries to make a change to your product screen and the tests fail and he comes over to you and asks “What now?” and you’re like, “duh, re-record those screenshots.” Then Alice tries to make another UI change, and has her tests fail, and asks you “What now?” and you’re like “duh, re-record those screenshots.” Eventually that happens enough times that you create a wiki page for it.
Once a week, you send that wiki page to a new-hire or intern.
But everyone knows you now, so that’s nice. You’re the screenshot testing person. They secretly hate you each time they re-record screenshots, but they certainly know you. And they’re also not going to add new screenshots, because each new screenshot means more screenshots for them to record.
That Linux person
So you’re basking in your new found fame, and along comes a new hire, with their disdain for corporate America and refusing to use the company Macs. Only Linux, thank-you-very-much. IT is annoyed but gets them a beefy Intel machine and tells them to do whatever with it.
So this person is chugging along, makes a UI change, and the usual drill happens: they message you, you introduce yourself (they need to know you!), you send them the wiki-page. They try to re-record and tell you that “the tests still don’t pass.”
So you dig around more Medium posts, and you find out that screenshot tests are dependent on the platform where you record screenshots. Linux and Mac, Intel and ARM, they generate slightly different screenshots.
You help them debug things for a while and eventually get mad, and tell them to get a Mac “like everyone else”. They refuse. A brawl ensues, HR is called in, your manager mediates and makes you compromise: it’s the screenshot tests or your job. You choose your job, the screenshot tests are disabled.
The DevOps person
But let’s say you did convince your Linux colleague to switch to a Mac. Things are going smoothly, until one day not long after, you have a DevOps engineer ping you, asking you why the repo size is growing so fast and accusingly points at all those PNG files.
“Oh,” you say “that’s the screenshot tests! super cool stuff, it lets… “ The DevOps engineer cuts you off and tells you you need to delete those screenshots. You propose this magic tool called GitLFS that all those Medium posts talk about, “could you help me set it up?”.
“Absolutely!”, says the DevOps engineer “Let me set up a meeting with the team, and prioritize this for Q3! anyway, thanks for understanding! In the meantime, let’s delete those screenshots.”
That Super-Senior-Staff Engineer
You know the one. This is the person that everybody looks up to. Their word is final. Everyone wants to be in their good grace for their promotion’s sake.
So this one time, their UI tests fail. Cue the annoyed back and forth with you, and you send them the wiki page etc.
“The screenshots look the same,” this person says.
You can’t argue of course, so you put everything you’re doing aside, and you pull up imagemagick and compare the images and show them what the difference is.
“Ok”, they say and moves along. Crisis averted. Now to figure out what you were working on before this interruption…
To summarize
Here’s what we learnt:
- You will be the screenshot testing person
- Each new dev needs to learn how to re-record screenshots
- Screenshot recording will depend on the machine being recorded
- At some point you will have to build and maintain a GitLFS infrastructure
- Image comparison can be annoying, and people will ping you to do the comparison for them because it was your tests that blocked them.
How do you solve this?
Most of these problems can be solved with a simple trick: Record your screenshots on CI. So usually when a company has invested enough time in screenshot testing, they’ll eventually create a CI job to re-record screenshots and update your PR with the updated screenshots. This avoids all the issues with devs manually re-recording screenshots, and it also solves the issues with platform dependence since you’re always recording on an identical CI machine.
(This isn’t trivial to build from scratch though, you also need to figure out a way for devs to locally run screenshot tests.)
You still have the DevOps overhead of maintaining GitLFS, and having to deal with subpar image comparison tools that GitHub provides.
Using an external Service
So companies like AirBnb and Facebook have gone a different direction: on each CI run, the screenshots are generated an uploaded to a different service that maintains the screenshots. (Note that it is easier to scale a service than GitLFS, since in most cases you don’t need to read the entire image from disk, you only need a hash of the file content.)
Most of these services will just notify you with a build status on GitHub showing you the changed screenshots, and giving you a chance to review them.
A service also can provide better image comparison tools. In fact Screenshotbot has some pretty fantastic image comparison tools, which let’s you zoom to pixel changes, show you exact RGB values of pixels that changed etc.
Screenshotbot does not use GitLFS, we store our images on our own infrastructure that you don’t have to worry about, so it also reduces the operational overhead and costs for you.
An external service is also super easy to get started with. For example, we have a Gradle plugin that makes it trivial on Android, and for other platforms you just need to pass the recorded screenshots to our command line tool. Our backend is open source, so you don’t need to get approvals or budget commitments to set up a server.
But most importantly: you’ll no longer be known as the screenshot testing person. People will get notified when screenshot tests change, they don’t need to re-record screenshots and they don’t need to look up wiki pages. Nobody will ping you to compare screenshots for them since Screenshotbot will do it for them.
But your company’s screenshot tests will explode in number since adding screenshots doesn’t slow developers down. You’ll be the Hero that nobody will ever know…, so make sure you brag about it and share metrics on your slack channel every quarter about how much your screenshots have grown. (We’re building an analytics page very soon just so you can do this.)
Screenshotbot Blog 