One day a handful of years ago, hanging out after work with coworkers, our typical anything-goes, no-holds-barred conversations turned to test frameworks:

“I don’t like any of our choices. I want tests to execute just like Ruby scripts do. I want variable scoping to just be the lexical scoping that’s already in Ruby. I want an assertion API that has two methods: assert and refute. I want the API to be so tight and focused that it’s practically impossible for major updates to break compatibility with our test code. I want to create test abstractions with the ease of mixing in a Ruby module into a Ruby class. And I don’t want a framework to be bloated with tools that the operating system already has.”

In 2015, we’d been Ruby devs for a decade. I’d been a user of test automation since 1997. Nathan started his professional life as an embedded chip test automator. We’d used RSpec, MiniTest, and Shoulda. Yada yada. But we’d grown disenchanted with the growing elaborateness of the APIs and the prescriptiveness of approaches that were parting ways with things we value. And we’d used TestUnit, but didn’t prefer it.

And like any sucker who’s ever found themselves entangled in a multi-year open source odyssey of alternating pain and suffering, Nathan Ladd suggested that it shouldn’t be too difficult to put something together over a long weekend that we could start to use in our work right away. Not wanting to squelch the unhinged optimism of someone who just might end up building something totally great in their spare time, we agreed.

Fast forward: June 2020. TestBench v1.0.0.0 is released. Nathan is expected to make a full recovery after a few more months of treatment.

I’ve worked with all the various incarnations of TestBench since it pulled itself out of the primordial soup of after work beer. Here are some of the things I love about it.

It’s Just Enough Test Framework

TestBench is the exact right fit for the kind of work I usually do.

I almost never work directly on Rails apps. Most of my work is on libraries and services that follow the RubyGems standards for project structure.

TestBench was built to give the Eventide Project control over its testing landscape. We also use it to build applications and evented services built on Eventide.

Over the years, users of Eventide have been exposed to TestBench and have adopted it for their own work.

In the coming months, we’ll be releasing some TestBench test fixture abstractions for Eventide users to make their testing experience even better.

But TestBench is absolutely not tied to Eventide. It’s just the right amount of test framework for the job.

The Core API is Compact

The core API has only a handful of methods: context, test, assert, refute, assert_raises, refute_raises, and comment.

Because it’s small and focused, it’s exceedingly unlikely that major updates will require any changes to existing test code.

The chance that this compact API will conflict with any of your own test code is also exceedingly small. It would require not only that you define methods in your test contexts, but that these methods would have the exact same name as the TestBench API methods.

Most of the time, only a subset of TestBench’s API is used.

context "Some Context" do
  test "Some test" do
    assert(true)
  end

  test "Some other test" do
    refute(false)
  end
end

More about the TestBench API at:
http://test-bench.software/user-guide/writing-tests.html

The Runner is Just Ruby

TestBench’s test runner is just the Ruby executable. If you’ve got the ruby executable in your machine’s search path (and you know you do), you already have the TestBench runner at your fingertips. And there’s no need to configure any special plugin for your editor.

> ruby test/automated/some_test.rb
Some Context
  Some test
  Some other test

However, TestBench also provides a more robust, dedicated runner that can be used in rare circumstances where more robust options are needed.

More about running tests at:
http://test-bench.software/user-guide/running-tests.html

Tests Are Ruby Scripts

Test files are Ruby scripts. The run like any Ruby script.

Test code is executed once, from top of a test script to the bottom.

There aren’t any affordances for setup and teardown, or any before and after blocks. Code at the top of a context can be used to set up state, and code at the bottom can be used to tear down state.

The code executes exactly the way it reads, without having to take into consideration any specialized runtime semantics when structuring and implementing tests.

It works the way Ruby works.

Variable Declaration and Scoping is Just Ruby

There’s no specialized syntax for variable declaration, and variable scoping is just Ruby’s lexical scoping.

context "Some Context" do
  context "Some Inner Context" do
    some_variable = 'some_value'

    context "Some Deeper Context" do
      puts some_variable
      # => "some_value"
    end
  end

  puts some_variable
  # => NameError (undefined local variable or method `some_variable' for main:Object)
end

Clean Test Design is Encouraged

It’s probably more accurate to say that clean test design is required.

TestBench doesn’t offer any of safety affordances that help keep new developers from common mistakes. It’s a sharp tool with sharp edges intended for developers who are comfortable running with scissors.

Test isolation is the responsibility of the developer rather than the framework.

Developers are expected to be in complete and total control over test state and not rely on any framework or test runner features to ensure proper test isolation. Test isolation is achieved by designing isolated tests. Ensuring that test files are singularly-focused and highly cohesive is the responsibility of the developer.

Using techniques like explaining variable are instrumental in leveraging TestBench’s minimal assertions API.

Test Abstractions Are Just Ruby Objects

Test fixture abstractions are just plain old Ruby objects.

By including the TestBench::Fixture module into a Ruby object, the object acquires all of the TestBench core API methods available to a test script, including context, test, assert, refute, assert_raises, refute_raises, and comment.

A test fixture can be used like any other Ruby object, including running it from within a test script, as well as testing the fixture itself.

class SomeFixture
  include TestBench::Fixture

  attr_accessor :something
  attr_accessor :something_else

  def initialize(something, something_else)
    @something = something
    @something_else = something_else
  end

  def call
    context "Some Context" do
      context "Something" do
        included = something_else.include?(something)

        test "Included in Something Else" do
          assert(included)
        end
      end

      context "Something Else" do
        twice_as_long = something_else.length == something.length * 2

        test "Twice as long as something" do
          assert(twice_as_long)
        end
      end
    end
  end
end

TestBench’s fixture method is a shortcut that initializes a fixture class, integrates it with TestBench’s output printer, and runs the fixture in the context of the current test script.

context "Other Context" do
  something = 'some value'
  something_else = something * 2

  fixture(SomeFixture, something, something_else)
end

More about fixture abstractions at:
http://test-bench.software/user-guide/fixtures.html

The TestBench API Will Always Be Backward Compatible

Because the TestBench API is so compact, there’s very little that could possibly change. Future releases will not need to break backward compatibility.

I can’t say how frustrating it is for a test framework to release a new version and be forced to rework the existing test suites because the new version isn’t compatible with existing tests. And worse, when a version of Rails is released that requires a major update to the test framework which in turn requires reworking all of the tests. Not to mention the broken compatibility with editor plugins.

This kind of problem is the result of overly-elaborate tools that are highly-prone to change over time. TestBench avoids the whole problem by staying focused on its core mission and compact core API.

No Features That Can Already Be Handled by the Operating System

TestBench specifically avoids any features that can already be handled by the operating system.

For example, the operating system is already capable of randomizing the order of test file execution. And because test files are focused and cohesive by design, randomization can be safely simplified to work at the level of files rather than individual test blocks.

Parallelization of test execution is also something that the operating system is already capable of, and TestBench leaves it to the operating system to handle this kind of thing.

As a bonus, by allowing the operating system to do the things that the operating system is already good at, the user gets to learn more about the operating systems, shell scripting, and common Unix and Linux utilities.

More about using the operating system and other recipes at:
http://test-bench.software/user-guide/recipes.html

No Forced Compliance with Opinions and No Shaming

TestBench doesn’t shy away from opinions, but it’s committed to rooting its opinions in fundamental design principles.

TestBench doesn’t restrict developers’ freedom to avoid clean and principled test design. Instead it provides constraints that require developers to take a principled stance to test design. A developer is free to be as diligent as they choose and make the tradeoffs they wish to make, knowing the risks.

The only strict, orthodox stance that TestBench takes is in regard to design principles. As long as design principles are respected, the outcomes will be just fine.

But TestBench doesn’t force its users into subservience, or worse, force users to debase themselves when they go against the doctrinal edicts of the tool’s developers. For example, if a developer faces some circumstance unforeseen by the framework’s designers, the framework doesn’t require users to shame themselves by invoking methods that start with i_suck, as we’ve seen with Ruby test frameworks that can be careless with user’s state of mind.

There’s already enough shaming and bullying in developer culture. We invite developers to go deeper into their journey into software design, but we don’t seek to harm them for not hitting a design homerun on their first attempts.

Thoughtful Community

And last but certainly not least, TestBech has an awesome, thoughtful, studious community that is invested in always learning more and experiencing software design and software testing.

Since TestBench grew out of the Eventide Project, the TestBench community is largely the Eventide community, and all Eventide users are well-versed in TestBench.

Interact with the community on the #test-bench channel of the Eventide Slack: eventide-project-slack.herokuapp.com

Amongst Others

There are many more subtle and thoughtful features of TestBench that make me a bit giddy when I encounter them, but this is intended to be a quick summary. I look forward to hearing more from others as they take their first steps with TestBench. Jump in and join the community and the conversation!