ruby

The Hidden Complexities of Running Shell Commands in Ruby

Did you know that system supports the equivalent of > dev/null output redirection? And that backticks raise an exception when a command isn’t found, but system does not? Let’s go on a deep dive into the Ruby standard library!

What’s the big deal?

It’s easy to run shell commands from Ruby, right?

system "bundle install"
This starts a sub-shell running the bundle install command, and waits for it to complete. While running, the command’s stdout and stderr will be streamed to the console.

The devil is in the details though. Consider these edge cases:

  • What happens if bundle install fails? How can we ensure an exception is raised?
  • What if we want to hide Bundler’s output from being printed to the console?
  • What if we want to capture what Bundler printed to stdout? Or stderr?
  • How about streaming Bundler’s output to the console so the user can watch the progress, and capturing it for later processing?

Turns out, system can do some – but not all – of those things. For the remaining cases, Ruby provides backticks (`) and the open3 module. Let’s dig deeper into Ruby’s bag of tricks.

The hidden abilities of system

The great thing about Ruby’s standard library is that methods like system are intuitive and very approachable, but also have powerful capabilities for advanced users. Here are a few of the non-obvious things that system can do.

Raise an exception when something goes wrong

If something goes wrong when running a command, system will return false or nil. It’s easy to forget to check the return value, though.

Starting with Ruby 2.7, system supports the exception: true keyword argument. This comes in handy when you want a guarantee that a command ran successfully.

system "bundle install", exception: true
# => true

system "bundle uninstall", exception: true
# => Command failed with exit 15: bundle (RuntimeError)

system "bundl install", exception: true
# => No such file or directory - bundl (Errno::ENOENT)
When exception: true is specified, system will raise a RuntimeError if the command exits with a non-zero status code, or a subclass of SystemCallError if the command can’t be executed (e.g. command not found).

The bin/rails script included in Rails projects makes use of this technique to define a system! method that raises on failure.

Hide stdout and/or stderr output

Normally, the stdout and stderr of the command spawned by system will be connected to stdout and stderr of the Ruby process. That means all the output from system "bundle install", for example, will be streamed to the console.

You can redirect this output with plain Ruby by specifying the out: and err: arguments, which is great for hiding noisy/distracting console messages.

# bundle install > /dev/null
system "bundle install", out: File::NULL

# bundle install > /dev/null 2>&1
system "bundle install", out: File::NULL, err: File::NULL
This hides Bundler’s output by redirecting it to the special NULL file descriptor. It’s similar to how you would use the > /dev/null technique in bash, but in a shell/OS-agnostic way.

Access the exit status code

The return value of system is true, false, or sometimes nil, depending on whether the command succeeded or failed. If you need more information about the failure, you can use Process.last_status, or its shorthand, $?.

system "bundle uninstall"      # => false
Process.last_status.success?   # => false
Process.last_status.exitstatus # => 15
The spelling of the exitstatus method always trips me up. Shouldn’t there be an underscore in there? 🤨

The gotchas of using backticks

One thing that system cannot easily do is capture output. That’s where Ruby’s backticks come in (defined via Kernel#` method).

output = `bundle install`
# => "Fetching gem metadata from https://rubygems.org/...\nResolving dependencies..."
Process.last_status.success? # => true or false
The backticks return the command’s stdout output (not stderr) as a string. It’s up to you to explicitly check the exit status.

However, beware of these limitations:

  • Stderr is not captured. It will still be streamed to the console.
  • The captured stdout will no longer be streamed to the console. Often this is desirable, but for a potentially long-running command like bundle install, that means there could be a long pause with no visible feedback while Ruby is waiting for the command to finish.

Differences from system

Note that system and backticks behave differently when asked to execute a non-existent command:

system "bundl install"
# => nil

`bundl install`
# => No such file or directory - bundl (Errno::ENOENT)
Backticks will raise an exception if the executable cannot be loaded, whereas system just returns nil. (Tested on Ruby 3.2, macOS)

Like system, backticks will not raise an exception if the command returns a non-zero status:

system "bundle oops"
# => false

`bundle oops`
# => ""
Process.last_status.success?   # => false
Process.last_status.exitstatus # => 127
The bundle oops command runs, but returns a 127 exit status without printing anything to stdout. Backticks therefore return an empty string, since stdout was empty.

Bundler interference

Things can get a bit more complicated when the command you are running is itself another Ruby script. In a word: Bundler.

Bundler’s main job is to restrict what gems (and versions of those gems) your Ruby code is allowed to access. These restrictions pass down from parent process to child process via special environment variables that Bundler injects.

That means that if you use system or backticks in a Ruby file that is governed by Bundler (say, within a Rails app), the command that you run will not be able to access gems outside your Gemfile. You might get an error like this:

system "ruby my_script.rb"
# => kernel_require.rb:37:in `require': cannot load such file -- httparty (LoadError)
The my_script.rb file is trying to load the httparty gem, which is installed. But the Bundler environment from the parent process is preventing the child process from loading it.

Use Bundler.with_original_env

If you run into a situation where Bundler is interfering with your command, the solution is to use Bundler.with_original_env. This trick is fairly common for Ruby code that invokes other Ruby scripts. Rails, for example, uses this technique in its app generators and in spring.

Bundler.with_original_env do
  system "ruby my_script.rb"
end
This prevents Bundler from interfering with the gems that my_script.rb is allowed to access.

If your script is sometimes invoked via Bundler and sometimes not, you’ll need a bit more boilerplate:

def with_original_bundler_env(&block)
  return yield unless defined?(Bundler)

  Bundler.with_original_env(&block)
end

# Now this code will work with or without Bundler
with_original_bundler_env do
  system "ruby my_script.rb"
end
This helper method delegates to Bundler if it is loaded, otherwise it is a pass-thru.

Handling advanced use-cases with open3

Together, system and backticks handle most use cases of running shell commands from Ruby. For advanced use-cases, Ruby provides the open3 module.

Capturing stderr

Ruby’s backticks capture stdout only. To capture stderr, open3 offers the capture2e and capture3 methods.

require "open3"

# Capture stdout and stderr as a single string
out_and_err, status = Open3.capture2e("bundle install")

# Capture stdout and stderr separately
out, err, status = Open3.capture3("bundle install")

# Process.last_status does *not* work with open3
Process.last_status # => nil

# You need to use the status object from return value
status.success?   # => true
status.exitstatus # => 0
Open3’s capture methods do not raise exceptions for non-zero exits, but do return a status object that can be used to check for success or failure. Like backticks, the captured output is returned as a string and is not streamed to the console.

Streaming output while capturing it

When capturing output from a long-running command, it can be nice to stream the output to the console for user-feedback. This is possible using the block form of popen3, which gives you access to the raw IO of the child process while it is running.

Here’s an example from the vite_ruby source code:

def capture(*cmd, with_output: $stdout.method(:puts), stdin_data: '', **opts)
  return Open3.capture3(*cmd, **opts) unless with_output

  Open3.popen3(*cmd, **opts) { |stdin, stdout, stderr, wait_threads|
    stdin << stdin_data
    stdin.close
    out = Thread.new { read_lines(stdout, &with_output) }
    err = Thread.new { stderr.read }
    [out.value, err.value.to_s, wait_threads.value]
  }
end

def read_lines(io)
  buffer = +''
  while line = io.gets
    buffer << line
    yield line
  end
  buffer
end
This capture method starts a child process to run cmd using popen3. While the command is running, 2 threads are spawned to simultaneously read the stdout and stderr of the child process. The stdout data is accumulated in a string while also being printed to the console via puts.

Conclusion

On the one hand, running shell commands in Ruby is a breeze. The system and backticks methods are intuitive and you can do a lot with with very little code:

system("bundle check") || system("bundle install")
puts "This project has #{`bundle list`.scan(/^  \*/).count} gems!"
# => This project has 115 gems!
system returns a boolean, making it easy to chain commands with boolean logic. And the string captured by backticks has dozens of powerful methods for further processing command output.

The downside of this expressiveness is that error handling is pretty much non-existent, and the bundle check output that gets printed to the console is a little distracting.

If the script you’re building requires a bit more robustness, you could use some of the techniques from this article:

system("bundle check", out: File::NULL, err: File::NULL) \
  || system("bundle install", exception: true)

gem_count = `bundle list`.scan(/^  \*/).count
raise "Unable to run `bundle list`!" unless Process.last_status.success?

puts "This project has #{gem_count} gems!"
# => This project has 115 gems!
This version is not quite as nice to look at, but arguably provides a more robust user experience. It hides the redundant/misleading bundle check output, and halts execution if any of the commands unexpectedly fail to run.

References

For more background, check out these pages from the Ruby 3.2.2 API docs:

Share this? Copy link

Feedback? Email me!

Hi! 👋 I’m Matt Brictson, a software engineer in San Francisco. This site is my excuse to practice UI design, fuss over CSS, and share my interest in open source. I blog about Rails, design patterns, and other development topics.

Recent articles

RSS
View all posts →

Open source projects

mattbrictson/nextgen

Generate your next Rails app interactively! This template includes production-ready recommendations for testing, security, developer productivity, and modern frontends. Plus optional Vite support! ⚡️

58
Updated 7 days ago

mattbrictson/tomo

A friendly CLI for deploying Rails apps ✨

370
Updated 1 month ago

More on GitHub →