The Hidden Complexities of Running Shell Commands in Ruby

What’s the big deal?

It’s easy to run shell commands from Ruby, right?

system "bundle install"

The devil is in the details though. Consider these edge cases:

• What happens if bundle install fails? How can we ensure an exception is raised?
• What if we want to hide Bundler’s output from being printed to the console?
• What if we want to capture what Bundler printed to stdout? Or stderr?
• How about streaming Bundler’s output to the console so the user can watch the progress, and capturing it for later processing?

Turns out, system can do some – but not all – of those things. For the remaining cases, Ruby provides backticks (`) and the open3 module. Let’s dig deeper into Ruby’s bag of tricks.

• What’s the big deal?
• The hidden abilities of system
• The gotchas of using backticks
• Bundler interference
• Handling advanced use-cases with open3
• Conclusion
• References

The hidden abilities of system

The great thing about Ruby’s standard library is that methods like system are intuitive and very approachable, but also have powerful capabilities for advanced users. Here are a few of the non-obvious things that system can do.

Raise an exception when something goes wrong

If something goes wrong when running a command, system will return false or nil. It’s easy to forget to check the return value, though.

Starting with Ruby 2.7, system supports the exception: true keyword argument. This comes in handy when you want a guarantee that a command ran successfully.

system "bundle install", exception: true
# => true

system "bundle uninstall", exception: true
# => Command failed with exit 15: bundle (RuntimeError)

system "bundl install", exception: true
# => No such file or directory - bundl (Errno::ENOENT)

The bin/rails script included in Rails projects makes use of this technique to define a system! method that raises on failure.

Hide stdout and/or stderr output

Normally, the stdout and stderr of the command spawned by system will be connected to stdout and stderr of the Ruby process. That means all the output from system "bundle install", for example, will be streamed to the console.

You can redirect this output with plain Ruby by specifying the out: and err: arguments, which is great for hiding noisy/distracting console messages.

# bundle install > /dev/null
system "bundle install", out: File::NULL

# bundle install > /dev/null 2>&1
system "bundle install", out: File::NULL, err: File::NULL

Access the exit status code

The return value of system is true, false, or sometimes nil, depending on whether the command succeeded or failed. If you need more information about the failure, you can use Process.last_status, or its shorthand, $?.

system "bundle uninstall"      # => false
Process.last_status.success?   # => false
Process.last_status.exitstatus # => 15

The gotchas of using backticks

One thing that system cannot easily do is capture output. That’s where Ruby’s backticks come in (defined via Kernel#` method).

output = `bundle install`
# => "Fetching gem metadata from https://rubygems.org/...\nResolving dependencies..."
Process.last_status.success? # => true or false

However, beware of these limitations:

• Stderr is not captured. It will still be streamed to the console.
• The captured stdout will no longer be streamed to the console. Often this is desirable, but for a potentially long-running command like bundle install, that means there could be a long pause with no visible feedback while Ruby is waiting for the command to finish.

Differences from system

Note that system and backticks behave differently when asked to execute a non-existent command:

system "bundl install"
# => nil

`bundl install`
# => No such file or directory - bundl (Errno::ENOENT)

Like system, backticks will not raise an exception if the command returns a non-zero status:

system "bundle oops"
# => false

`bundle oops`
# => ""
Process.last_status.success?   # => false
Process.last_status.exitstatus # => 127

Bundler interference

Things can get a bit more complicated when the command you are running is itself another Ruby script. In a word: Bundler.

Bundler’s main job is to restrict what gems (and versions of those gems) your Ruby code is allowed to access. These restrictions pass down from parent process to child process via special environment variables that Bundler injects.

That means that if you use system or backticks in a Ruby file that is governed by Bundler (say, within a Rails app), the command that you run will not be able to access gems outside your Gemfile. You might get an error like this:

system "ruby my_script.rb"
# => kernel_require.rb:37:in `require': cannot load such file -- httparty (LoadError)

Use Bundler.with_original_env

If you run into a situation where Bundler is interfering with your command, the solution is to use Bundler.with_original_env. This trick is fairly common for Ruby code that invokes other Ruby scripts. Rails, for example, uses this technique in its app generators and in spring.

Bundler.with_original_env do
  system "ruby my_script.rb"
end

If your script is sometimes invoked via Bundler and sometimes not, you’ll need a bit more boilerplate:

def with_original_bundler_env(&block)
  return yield unless defined?(Bundler)

  Bundler.with_original_env(&block)
end

# Now this code will work with or without Bundler
with_original_bundler_env do
  system "ruby my_script.rb"
end

Handling advanced use-cases with open3

Together, system and backticks handle most use cases of running shell commands from Ruby. For advanced use-cases, Ruby provides the open3 module.

Capturing stderr

Ruby’s backticks capture stdout only. To capture stderr, open3 offers the capture2e and capture3 methods.

require "open3"

# Capture stdout and stderr as a single string
out_and_err, status = Open3.capture2e("bundle install")

# Capture stdout and stderr separately
out, err, status = Open3.capture3("bundle install")

# Process.last_status does *not* work with open3
Process.last_status # => nil

# You need to use the status object from return value
status.success?   # => true
status.exitstatus # => 0

Streaming output while capturing it

When capturing output from a long-running command, it can be nice to stream the output to the console for user-feedback. This is possible using the block form of popen3, which gives you access to the raw IO of the child process while it is running.

Here’s an example from the vite_ruby source code:

def capture(*cmd, with_output: $stdout.method(:puts), stdin_data: '', **opts)
  return Open3.capture3(*cmd, **opts) unless with_output

  Open3.popen3(*cmd, **opts) { |stdin, stdout, stderr, wait_threads|
    stdin << stdin_data
    stdin.close
    out = Thread.new { read_lines(stdout, &with_output) }
    err = Thread.new { stderr.read }
    [out.value, err.value.to_s, wait_threads.value]
  }
end

def read_lines(io)
  buffer = +''
  while line = io.gets
    buffer << line
    yield line
  end
  buffer
end

Conclusion

On the one hand, running shell commands in Ruby is a breeze. The system and backticks methods are intuitive and you can do a lot with with very little code:

system("bundle check") || system("bundle install")
puts "This project has #{`bundle list`.scan(/^  \*/).count} gems!"
# => This project has 115 gems!

The downside of this expressiveness is that error handling is pretty much non-existent, and the bundle check output that gets printed to the console is a little distracting.

If the script you’re building requires a bit more robustness, you could use some of the techniques from this article:

system("bundle check", out: File::NULL, err: File::NULL) \
  || system("bundle install", exception: true)

gem_count = `bundle list`.scan(/^  \*/).count
raise "Unable to run `bundle list`!" unless Process.last_status.success?

puts "This project has #{gem_count} gems!"
# => This project has 115 gems!

References

For more background, check out these pages from the Ruby 3.2.2 API docs:

• Open3
• Kernel#system
• Kernel#` (backticks)
• Kernel#spawn (explains advanced features supported by system and Open3, like environment variables, umask, and redirection)