How to Test Ruby Projects on Windows
AppVeyor is a continuous integration service like Travis CI, except that AppVeyor runs on Windows servers. I’ve found that it can be an easy way to test that a Ruby project works in a Windows environment. Here’s how to set it up.
Have you noticed that Ruby developers are overwhelmingly Linux and Mac users? Because of this collective blindspot, sometimes gems produced by the Ruby community fail to work properly on that other big platform: Windows. If you’re authoring a gem, setting up Windows-based continuous integration can help guard against this oversight.
AppVeyor, like Travis CI, is free for open source projects. Being Windows-based, AppVeyor’s audience is primarily .NET developers, and so Ruby documentation and examples seem to be hard to find. But running Ruby on AppVeyor is possible with these steps:
1. Set up a CI project
Visit appveyor.com and log in via GitHub. Once connected, create a project for the GitHub repository you wish to test. AppVeyor will configure webhooks to trigger a build on your next commit.

appveyor.yml
file. This will keep AppVeyor builds from failing until things are properly configured.2. Create appveyor.yml
AppVeyor uses an appveyor.yml
file at the root of a project to configure the build process (the concept is the same as .travis.yml
, but note there isn’t a leading .
).
Through some trial and error, I’ve figured out the basic appveyor.yml
that works for building a Ruby gem project:
version: '{build}'
skip_tags: true
environment:
matrix:
- ruby_version: "21"
- ruby_version: "21-x64"
install:
- SET PATH=C:\Ruby%ruby_version%\bin;%PATH%
- gem install bundler --no-document -v 1.10.5
- bundle install --retry=3
test_script:
- bundle exec rake
build: off
Most of this is self-explanatory, but there are a few options that are a bit tricky:
ruby_version
Note that Ruby 2.1.x is specified as "21"
and "21-x64"
for the 64-bit version. Ruby 2.2.x is also available on AppVeyor, but in my experience it does not work reliably.
test_script
These are the commands to actually execute the test suite. Typically for Ruby projects this is bundle exec rake
.
build
Out of the box, AppVeyor seems to assume that you will be building a .NET project, and will fail if you are testing something different. For Ruby, disable this behavior with build: off
.
3. Commit and troubleshoot
When you commit appveyor.yml
and push it to GitHub, this will automatically trigger a build. Keep in mind that AppVeyor builds in their free open source tier are often queued for several minutes before they run. Be patient!

With luck, your first build will execute without errors. Mine, unfortunately, did not. But fixing things was pretty straightforward. Here are some of the issues I encountered:
- On AppVeyor,
$stdout.tty?
returnsfalse
, whereas on Travis it istrue
. This may cause trouble if you are testing code that behaves differently based ontty?
. - There is no such thing as
> /dev/null
on Windows, so running a command while muting its output usingsystem("some command > /dev/null")
won’t work. One alternative is`some command`
, which captures the output; then testing the result with$?.success?
. Etc.getpwuid
returnsnil
on Windows. I’m guessing the entireEtc
module probably doesn’t work.- On Windows, the block form of
Dir.mktmpdir
(which is supposed to clean up after itself) failed for me with mysteriouspermission denied
errors. I had better luck using the non-block form and cleaning up manually usingFileUtils.rm_rf
. - For some reason the
gem install bundler
step tends to get stuck during AppVeyor builds, sometimes for several minutes. I couldn’t find a solution except to wait patiently for it to get un-stuck.
It works!
I’m happy to report that two of my open source projects, airbrussh and chandler, both have successful Windows builds. Even though I don’t use Windows myself, thanks to the continuous integration I now have some peace of mind that I am not putting Windows-incompatible code out into the world.
By the way, the Travis team also has Windows support on its radar, so eventually we may have a single tool that can support all the platforms we need. Until then AppVeyor is a good option and doesn’t take too much work to set up.
Have you used AppVeyor to build a Ruby project? How did it go?