How to Test Ruby Projects on Windows

AppVeyor is a continuous integration service like Travis CI, except that AppVeyor runs on Windows servers. I’ve found that it can be an easy way to test that a Ruby project works in a Windows environment. Here's how to set it up.

Have you noticed that Ruby developers are overwhelmingly Linux and Mac users? Because of this collective blindspot, sometimes gems produced by the Ruby community fail to work properly on that other big platform: Windows. If you’re authoring a gem, setting up Windows-based continuous integration can help guard against this oversight.

AppVeyor, like Travis CI, is free for open source projects. Being Windows-based, AppVeyor’s audience is primarily .NET developers, and so Ruby documentation and examples seem to be hard to find. But running Ruby on AppVeyor is possible with these steps:

1 Set up a CI project

Visit and log in via GitHub. Once connected, create a project for the GitHub repository you wish to test. AppVeyor will configure webhooks to trigger a build on your next commit.

You may also want to navigate to the Settings and disable builds for branches that don’t have an appveyor.yml file. This will keep AppVeyor builds from failing until things are properly configured.

2 Create appveyor.yml

AppVeyor uses an appveyor.yml file at the root of a project to configure the build process (the concept is the same as .travis.yml, but note there isn’t a leading .).

Through some trial and error, I’ve figured out the basic appveyor.yml that works for building a Ruby gem project:

version: '{build}'

skip_tags: true

    - ruby_version: "21"
    - ruby_version: "21-x64"

  - SET PATH=C:\Ruby%ruby_version%\bin;%PATH%
  - gem install bundler --no-document -v 1.10.5
  - bundle install --retry=3

  - bundle exec rake

build: off

Most of this is self-explanatory, but there are a few options that are a bit tricky:

ruby_version. Note that Ruby 2.1.x is specified as "21" and "21-x64" for the 64-bit version. Ruby 2.2.x is also available on AppVeyor, but in my experience it does not work reliably.

test_script. These are the commands to actually execute the test suite. Typically for Ruby projects this is bundle exec rake.

build. Out of the box, AppVeyor seems to assume that you will be building a .NET project, and will fail if you are testing something different. For Ruby, disable this behavior with build: off.

3 Commit and troubleshoot

When you commit appveyor.yml and push it to GitHub, this will automatically trigger a build. Keep in mind that AppVeyor builds in their free open source tier are often queued for several minutes before they run. Be patient!

Build results appear in the AppVeyor UI in realtime.

With luck, your first build will execute without errors. Mine, unfortunately, did not. But fixing things was pretty straightforward. Here are some of the issues I encountered:

It works!

I’m happy to report that two of my open source projects, airbrussh and chandler, both have successful Windows builds. Even though I don’t use Windows myself, thanks to the continuous integration I now have some peace of mind that I am not putting Windows-incompatible code out into the world.

By the way, the Travis team also has Windows support on its radar, so eventually we may have a single tool that can support all the platforms we need. Until then AppVeyor is a good option and doesn’t take too much work to set up.

Have you used AppVeyor to build a Ruby project? How did it go?