Hello? Is anybody in there? Just nod if you can hear me...

For the past while, I've been writing a lot of tests to make up for the lack of them in some apps at work. Most of that time has been writing feature tests. I've encountered a lot of weird little things that I've struggled with. I don't quite know why some things have happened, but I've figured out how to get around or solve the problems I've been facing.

So this is all about the tips and tricks I've learned while writing feature tests.

But first, a digression

When I first learned Rails, I learned Behavior Driven Development (BDD) with Cucumber. If you're not aware, Cucumber isn't just a vegetable anymore! Cucumber tests use a DSL called Gherkin, which is business-readable. (Tho, what's up with the pickling vegetable names?) The concept behind it is that it's for the client to read and understand, as if they were created from the 3x5 index cards used to define the feature at the beginning. The format is set in a 'Given-When-Then' setup, such as: 'Given I have some chocolate, when I open the package, then it is eaten.' What was familiar English on the front, was actually a bunch of regex in the back combined with Capybara to drive the actions.

One of the common problems I've faced when using Cucumber was that the steps, the given/when/thens, would sometimes get repeated, not redundant, but slightly different in wording gave it its own step definition. Bloated. It was also annoying to try to find a step definition when one would say "When I click on a button" and another would say "When I click a button". This would be even more accentuated if there were multiple devs involved in the project writing their own step definitions. I wonder if this was one of the reasons why people stopped using cukes for testing and started incorporating feature tests with RSpec.

I, for one, have embraced using RSpec feature tests with Capybara. You still have the same expectations but without the cucumbersome extra layers of Gherkin (which no client actually really reads) and bloated step definitions. Instead, it's straight up RSpec and Capybara for you and your dev team.

Tips & Tricks

0.5) Save and open page

I'm always a bit surprised when I hear that devs have never heard of save_and_open_page, a Capybara method that lets you view a simple html-only snapshot of the page at invocation. Unfortunately it renders the page without the benefit of JS so things the javascript would hide would be visible and the angular interpolations would display. As long as if you're aware of the pitfalls, save_and_open_page is a fantastic lil line that can help you troubleshoot. Stick it in the feature spec anywhere before your expectation. You may need the launchy gem for this now.

1) Would you like some Javascript with those features?

Typically, our apps would have @javascript at the beginning of each scenario when we wanted to test a feature that incorporates Javascript, such as Angular. I am pretty sure this is a carry-over from the Cucumber days but in RSpec land, you can get away with simply stating js: true in your scenario, context, or even feature block.

As a lot of our apps use Angular, it's easier for me to put the js: true in the feature block. Here's what I mean.

require 'rails_helper'

feature 'Admin home page', :devise, js: true do
  # and then continue with your scenario blocks, before blocks, and whatever else
end

This gives for cleaner code, since each of your scenarios aren't littered with @javascript or tagged with js: true. This really comes in handy if your front-end app is fairly JS heavy.

2) Check response headers for downloads

One of our apps has a download function. I didn't want to test the downloaded file, but simply that the download actually worked. Instead of creating a file and comparing that against the downloaded copy, I "cheated" a bit and decided to check the response headers. This StackOverflow answer actually helped me out by pointing me in the direction I needed to go.

scenario 'it downloads a file', js: false do
  visit download_path

  click_on 'Download'
  expect(page.response_headers['Content-Type']).to eq 'image/jpeg'
end

As you can see, I also had to turn off Javascript by inserting js: false just for that test since the entire feature had Javascript turned on. This has to do with Capybara using Selenium (our default) to test the Javascript. Instead, having js: false forced Capybara to use its natural default of RackTest. Note that this makes the particular test go faster since it doesn't render Javascript.

3) Responsive web design and continuous integration

OK. Story time. For one of our apps, I had an awful time troubleshooting failed tests on our CI environment -- TravisCI. For whatever bizarre reason, certain tests would fail on Travis but never locally. I couldn't reproduce it at all, but the errors would almost always be about Capybara saying an element couldn't be found.

Failure/Error: click_on 'Download'

Capybara::ElementNotFound:
  Unable to find link or button "Download"

Annoying. I know. I'm looking straight at the page and saying "but there IS a download button! It's right there!"

I tried to add ids; I tried renaming it; I tried a slew of different things, all using save_and_open_page to see what the state of the page was. Without fail, "Download" appeared. As I surfed around the web, I had an epiphany. What if the test is being run in a smaller browser window? I didn't know what size "smaller" would be. I used Chrome's mobile-mode to test my theory and BOOM! The buttons I had been looking for had disappeared when the browser was at around 400px. This explained why my tests would fail. Our app uses responsive web design with bootstrap, so css values like hidden-lg or visible-lg were being used. Was Travis running my tests at mobile size (e.g. 400-ish pixels)?

The real question is, how do I get Travis to run my tests at a particular size? After some sleuthing, I found out that by default, Travis runs Firefox at around 1024x768. Ok. I tested whether or not I could see my buttons at 1024. Yup, the hamburglar menu popped up at that width and my "Download" button disappeared. Travis has in their docs that the browser size can be resized to whatever dimensions you want. I added the following before_install section to my .travis.yml file, keeping the xvbf start in the before_script. Note: Travis docs says having xvfb start doesn't play well w/ the custom dimensions but I haven't seen a problem with it yet.

before_install:
- "/sbin/start-stop-daemon --start --quiet --pidfile /tmp/custom_xvfb_99.pid --make-pidfile --background --exec /usr/bin/Xvfb -- :99 -ac -screen 0 1280x1024x16"


before_script:
- export DISPLAY=:99.0
- sh -e /etc/init.d/xvfb start

Tests were rerun, but they still failed. WTF, mate? Was my setting not being accepted? I surfed the web for something. Anything. I even posted on StackOverflow. Finally after about a week, I found the following article: A circleCI conversation about Capybara testing which included an interesting article called Testing responsive layouts. Voila!

The two articles gave me all the information I needed to proceed. Selenium needed a browser window size setting, so I simply gave it one. In my spec/spec_helper.rb I added the following:

def set_selenium_window_size_to_large
  page.driver.browser.manage.window.resize_to(1280, 1280)
end

def set_selenium_window_size_to_small
  page.driver.browser.manage.window.resize_to(600, 600)
end

Now I could simply call set_selenium_window_size_to_large (or small) from within my feature tests.

And Travis? Travis was super happy to give me some green builds after that. grinning

4) Refreshing the page

With the use of Angular, I've noticed some finickiness with some of my other tests. Once I solved the responsive-layout problem above, I encountered another issue of some tests failing, both locally and on Travis.

In this particular instance, it was another Capybara::ElementNotFound error, but not associated with the browser resolution problem above. The error was triggered only on certain instances. An example test that failed:

scenario 'editing an answer' do
  click_on 'Edit'

  fill_in 'Answer', with: 'something else'
  click_on 'Save'
  expect(page).to have_content 'something else'
end

The page uses Angular to display a modal/overlay that allows you to edit the answer, then saves that upon close. You could then click on 'Edit' to have the overlay pop up again to confirm the change. Unfortunately the tests did not like this. I suspect it was because the answer's edit page was in a tabbed view. I had to actually force the page to refresh in order to get the page overlay to display once more. So how do I refresh the page without actually saying visit such_and_such_path? Selenium had the answer.

page.driver.browser.navigate.refresh

This does mean that I have to add in a couple more extra clicks, but it made the headache go away.

5) Hunting down flaky tests

A couple of apps were failing their tests periodically: flaky tests. It seemed random. Underneath it all, there were 2 issues.

First one. Angular was too fast. During the scenario build, the transaction to create data wasn't finished when Angular asked for that data. This caused Angular to get back an empty data set causing the test to fail. No good. A quick, but brittle, fix to this was to insert a sleep 1 in the test. After speaking to a senior dev on the team, he sold me on the fact that having to rely on sleep could mean that there is actually something wrong with the code. For example, the db transaction could be taking too long because it's trying to do multiple things at once. If you have the time to investigate further, that would be ideal. Unfortunately there were not enough client hours to justify spending it to solve this issue so sleep 1 it was.

Second one. This one is a big mindboggling for me, but probably because I haven't figured out why it happened in the first place. We originally were running our cukes in headless mode (aka, no firefox browser popping up). It's faster and uses PhantomJS. However, everyone's environment seemed to react differently. Some folks had perfectly passing tests while others didn't. The environments should have been the same but were not. Worse was that the Gemfile.locks were all the same so it wasn't a versioning issue either. I dared to actually try to build a Docker dev environment for everyone to use. At least with that, everyone's environment would be the same. While setting this up, my coworker and I couldn't get PhantomJS to load or run properly with Docker, so we opted to use Selenium instead. On a whim, we ran our tests with Selenium. Everything passed! And despite my disdain for Docker, thank you! Because had we not gone that route and encountered obstacles, we wouldn't have switched to Selenium and fixed our build.

Ultimately, these might be common tips and tricks for feature tests, but I hope this helps anyone who is rather rusty or new to the game. Of course as I write this, I'm wondering if xvfb's screen size change is for "monitors" vs "browsers"......doh! sweat_smile

Level up +5

Questions? Comments? Hit me up at risaonrails !