Mad About Software

I Love TestBench

2020-06-11T12:00:00+00:00

One day a handful of years ago, hanging out after work with coworkers, our typical anything-goes, no-holds-barred conversations turned to test frameworks:

“I don’t like any of our choices. I want tests to execute just like Ruby scripts do. I want variable scoping to just be the lexical scoping that’s already in Ruby. I want an assertion API that has two methods: assert and refute. I want the API to be so tight and focused that it’s practically impossible for major updates to break compatibility with our test code. I want to create test abstractions with the ease of mixing in a Ruby module into a Ruby class. And I don’t want a framework to be bloated with tools that the operating system already has.”

In 2015, we’d been Ruby devs for a decade. I’d been a user of test automation since 1997. Nathan started his professional life as an embedded chip test automator. We’d used RSpec, MiniTest, and Shoulda. Yada yada. But we’d grown disenchanted with the growing elaborateness of the APIs and the prescriptiveness of approaches that were parting ways with things we value. And we’d used TestUnit, but didn’t prefer it.

And like any sucker who’s ever found themselves entangled in a multi-year open source odyssey of alternating pain and suffering, Nathan Ladd suggested that it shouldn’t be too difficult to put something together over a long weekend that we could start to use in our work right away. Not wanting to squelch the unhinged optimism of someone who just might end up building something totally great in their spare time, we agreed.

Fast forward: June 2020. TestBench v1.0.0.0 is released. Nathan is expected to make a full recovery after a few more months of treatment.

I’ve worked with all the various incarnations of TestBench since it pulled itself out of the primordial soup of after work beer. Here are some of the things I love about it.

It’s Just Enough Test Framework

TestBench is the exact right fit for the kind of work I usually do.

I almost never work directly on Rails apps. Most of my work is on libraries and services that follow the RubyGems standards for project structure.

TestBench was built to give the Eventide Project control over its testing landscape. We also use it to build applications and evented services built on Eventide.

Over the years, users of Eventide have been exposed to TestBench and have adopted it for their own work.

In the coming months, we’ll be releasing some TestBench test fixture abstractions for Eventide users to make their testing experience even better.

But TestBench is absolutely not tied to Eventide. It’s just the right amount of test framework for the job.

The Core API is Compact

The core API has only a handful of methods: context, test, assert, refute, assert_raises, refute_raises, and comment.

Because it’s small and focused, it’s exceedingly unlikely that major updates will require any changes to existing test code.

The chance that this compact API will conflict with any of your own test code is also exceedingly small. It would require not only that you define methods in your test contexts, but that these methods would have the exact same name as the TestBench API methods.

Most of the time, only a subset of TestBench’s API is used.

context "Some Context" do
  test "Some test" do
    assert(true)
  end

  test "Some other test" do
    refute(false)
  end
end

More about the TestBench API at:
http://test-bench.software/user-guide/writing-tests.html

The Runner is Just Ruby

TestBench’s test runner is just the Ruby executable. If you’ve got the ruby executable in your machine’s search path (and you know you do), you already have the TestBench runner at your fingertips. And there’s no need to configure any special plugin for your editor.

> ruby test/automated/some_test.rb
Some Context
  Some test
  Some other test

However, TestBench also provides a more robust, dedicated runner that can be used in rare circumstances where more robust options are needed.

More about running tests at:
http://test-bench.software/user-guide/running-tests.html

Tests Are Ruby Scripts

Test files are Ruby scripts. The run like any Ruby script.

Test code is executed once, from top of a test script to the bottom.

There aren’t any affordances for setup and teardown, or any before and after blocks. Code at the top of a context can be used to set up state, and code at the bottom can be used to tear down state.

The code executes exactly the way it reads, without having to take into consideration any specialized runtime semantics when structuring and implementing tests.

It works the way Ruby works.

Variable Declaration and Scoping is Just Ruby

There’s no specialized syntax for variable declaration, and variable scoping is just Ruby’s lexical scoping.

context "Some Context" do
  context "Some Inner Context" do
    some_variable = 'some_value'

    context "Some Deeper Context" do
      puts some_variable
      # => "some_value"
    end
  end

  puts some_variable
  # => NameError (undefined local variable or method `some_variable' for main:Object)
end

Clean Test Design is Encouraged

It’s probably more accurate to say that clean test design is required.

TestBench doesn’t offer any of safety affordances that help keep new developers from common mistakes. It’s a sharp tool with sharp edges intended for developers who are comfortable running with scissors.

Test isolation is the responsibility of the developer rather than the framework.

Developers are expected to be in complete and total control over test state and not rely on any framework or test runner features to ensure proper test isolation. Test isolation is achieved by designing isolated tests. Ensuring that test files are singularly-focused and highly cohesive is the responsibility of the developer.

Using techniques like explaining variable are instrumental in leveraging TestBench’s minimal assertions API.

Test Abstractions Are Just Ruby Objects

Test fixture abstractions are just plain old Ruby objects.

By including the TestBench::Fixture module into a Ruby object, the object acquires all of the TestBench core API methods available to a test script, including context, test, assert, refute, assert_raises, refute_raises, and comment.

A test fixture can be used like any other Ruby object, including running it from within a test script, as well as testing the fixture itself.

class SomeFixture
  include TestBench::Fixture

  attr_accessor :something
  attr_accessor :something_else

  def initialize(something, something_else)
    @something = something
    @something_else = something_else
  end

  def call
    context "Some Context" do
      context "Something" do
        included = something_else.include?(something)

        test "Included in Something Else" do
          assert(included)
        end
      end

      context "Something Else" do
        twice_as_long = something_else.length == something.length * 2

        test "Twice as long as something" do
          assert(twice_as_long)
        end
      end
    end
  end
end

TestBench’s fixture method is a shortcut that initializes a fixture class, integrates it with TestBench’s output printer, and runs the fixture in the context of the current test script.

context "Other Context" do
  something = 'some value'
  something_else = something * 2

  fixture(SomeFixture, something, something_else)
end

More about fixture abstractions at:
http://test-bench.software/user-guide/fixtures.html

The TestBench API Will Always Be Backward Compatible

Because the TestBench API is so compact, there’s very little that could possibly change. Future releases will not need to break backward compatibility.

I can’t say how frustrating it is for a test framework to release a new version and be forced to rework the existing test suites because the new version isn’t compatible with existing tests. And worse, when a version of Rails is released that requires a major update to the test framework which in turn requires reworking all of the tests. Not to mention the broken compatibility with editor plugins.

This kind of problem is the result of overly-elaborate tools that are highly-prone to change over time. TestBench avoids the whole problem by staying focused on its core mission and compact core API.

No Features That Can Already Be Handled by the Operating System

TestBench specifically avoids any features that can already be handled by the operating system.

For example, the operating system is already capable of randomizing the order of test file execution. And because test files are focused and cohesive by design, randomization can be safely simplified to work at the level of files rather than individual test blocks.

Parallelization of test execution is also something that the operating system is already capable of, and TestBench leaves it to the operating system to handle this kind of thing.

As a bonus, by allowing the operating system to do the things that the operating system is already good at, the user gets to learn more about the operating systems, shell scripting, and common Unix and Linux utilities.

More about using the operating system and other recipes at:
http://test-bench.software/user-guide/recipes.html

No Forced Compliance with Opinions and No Shaming

TestBench doesn’t shy away from opinions, but it’s committed to rooting its opinions in fundamental design principles.

TestBench doesn’t restrict developers’ freedom to avoid clean and principled test design. Instead it provides constraints that require developers to take a principled stance to test design. A developer is free to be as diligent as they choose and make the tradeoffs they wish to make, knowing the risks.

The only strict, orthodox stance that TestBench takes is in regard to design principles. As long as design principles are respected, the outcomes will be just fine.

But TestBench doesn’t force its users into subservience, or worse, force users to debase themselves when they go against the doctrinal edicts of the tool’s developers. For example, if a developer faces some circumstance unforeseen by the framework’s designers, the framework doesn’t require users to shame themselves by invoking methods that start with i_suck, as we’ve seen with Ruby test frameworks that can be careless with user’s state of mind.

There’s already enough shaming and bullying in developer culture. We invite developers to go deeper into their journey into software design, but we don’t seek to harm them for not hitting a design homerun on their first attempts.

Thoughtful Community

And last but certainly not least, TestBech has an awesome, thoughtful, studious community that is invested in always learning more and experiencing software design and software testing.

Since TestBench grew out of the Eventide Project, the TestBench community is largely the Eventide community, and all Eventide users are well-versed in TestBench.

Interact with the community on the #test-bench channel of the Eventide Slack: eventide-project-slack.herokuapp.com

Amongst Others

There are many more subtle and thoughtful features of TestBench that make me a bit giddy when I encounter them, but this is intended to be a quick summary. I look forward to hearing more from others as they take their first steps with TestBench. Jump in and join the community and the conversation!

Story Points Are Obsolete: Use Time Range Estimates Instead

2019-10-21T00:00:00+00:00

We started doing point estimates 20 years ago in response to conditions of software projects in the late 90s. Arguably those conditions haven’t existed for many years. And like continuing to use crutches longer than needed to recover from an injury, continuing to use point estimates creates new weaknesses and obscures approaches to estimation that could make us even stronger and healthier.

Why We Use Point Estimates

Point estimates serve two critical purposes: they provide a means for software developers to deal with uncertainty, and they firewall a software team from uninformed deadlines dictated from outside the group of people entrusted to do the actual software work.

Dealing With Dictated Estimates

The firewalling aspect of point estimates works by creating a unit of measure for software estimates that is completely incompatible with the standard units of measure used in every other aspect of the business: time and money.

Without a standard unit of measure to express expectations, software developers effectively disrupted the lines of communication that carried the most unrealistic, ill-conceived, and uninformed work estimate dictates.

By adopting units of measure like story points and t-shirt sizes that aren’t directly convertible to time and money, budget and schedule controllers and software workers no longer spoke the same language as the rest of the business.

To a great extent, the adoption of incompatible units of measure took some of the more unreasonable heat off of software teams and gave them a bit of a reprieve to repair some of the historical damage done by the more harmful aspects of the prescriptive methods of the 90s.

Dealing With Uncertainty

Point estimates’ ability to encode uncertainty into the estimates remains an essential benefit of point estimation.

Higher values in point estimation systems, whether using the Fibonacci scale or the t-shirt size scale, convey greater uncertainty.

For example, an estimate of 1 point inherently has much higher assurance than an estimate of 13 points, and an estimate of 13 points has much higher assurance than an estimate of 40.

An estimate of 1 point conveys certainty because the task is small, has fewer factors to consider, and therefore is much easier to predict. An estimate of 20 conveys a scale of work that leaves a lot of room for the unforeseen. Estimates made at the upper end of the scale are effectively conveying guesswork.

The same can be said of the t-shirt scale. An estimate of Small has higher assurance than an estimate of Extra Large.

Communicating uncertainty to project stakeholders and principals is critical feedback from the people doing the work to the people requesting that the work be done (not to mention having authority over budget).

Hindering Communication with Point Estimates

In the levels of the organization immediately above the software development team, everybody deals with business-standard units of measure: time and money.

When software teams use points for estimation and progress measurement, they force everybody else to do the translation.

That’s not the best basis for a cooperative relationship. It risks eroding the credibility of software developers.

Developers also, ironically, perform the same mental gymnastics when using story points and t-shirt sizes, converting them implicitly into time. Sprints and iterations are measured in time, while work is measured in points. We inevitably retrofit our point estimates into the fixed time box allocated by our chosen iteration length to check whether we’ve done enough estimating for the sprint or iteration.

The Solution: Just Estimate in Time (But Use Ranges of Time)

Instead of estimating in proxy tokens like points, we can just do the estimate in units of time.

One caveat: time estimates should be given in a range rather than a single data point.

The purpose of estimating in time ranges is to preserve the level of uncertainty inherent in an estimate. You’re still required to convey uncertainty to stakeholders and to team mates. And you still have an inalienable right to be truthful about the uncertainty of any work item, and to not be pressured into giving a false estimate.

Examples of Time Range Estimates

If a 3-point work item implicitly equates to 3 days, you could then just give the estimate as 3 days.

But provide 2 data points: an optimistic estimate and a pessimistic estimate.

For example, that 3-point work item might be 2 1/2 days under ideal conditions, and may be up to 5 days if that code that needs to be changed is affected by that coupling that keeps us from making changes in that part of the system.

The estimate is:

optimistic | pessimistic
------------------------
   2 1/2   |    6

The optimistic estimate represents a best-case scenario estimate. The pessimistic estimate represents a contingency for potential setbacks that we suspect are within the realm of possibility.

A Single Number Time Estimate Is a Promise Rather than an Estimate

Unless you’re absolutely, positively certain of your estimate, don’t provide it using a single number (aka: a single-point estimate).

You’re absolutely within your rights to provide a single-number estimate, but you’d better not miss the target.

Providing an estimate without a range is a promise, not an estimate. It conveys that you’re so certain of the estimate that you don’t need to convey any uncertainty at all.

There are times when a single-pointed estimate is appropriate. But just make sure that when you give a single-pointed estimate that you’re comfortable with the repercussions of not fulfilling that promise.

Don’t make a promise you can’t keep. Especially, don’t unintentionally give a promise when you meant to give an estimate.

Encoding Uncertainty in Time Range Estimates

The broader the range of the estimate, the more uncertainty that is expressed.

For example, an estimate of between a 1/2 day and 1 day has a good deal of certainty. But an estimate of between 1 day and 10 days expresses the potential for quite a good number of uncontrolled circumstances that can result in a 1-day work item needing to take two work weeks.

It’s entirely legitimate to submit an estimate for a work item that is between 0 days and an infinite number of days. Such an estimate simply says, “I just don’t know”, or “I don’t know yet”.

As time goes on and uncertainty is turned into more certainty with the emergence of new knowledge and learning, the estimates are refined to reflect the greater certainty that knowledge provides.

Decoupling Certainty from Size

Range estimates decouple of the estimated size of a work item from the certainty (or uncertainty) of the work item.

Big work items don’t necessarily have to convey uncertainty as often becomes the way when using point estimate schemes.

For example, an estimate of between 20 days and 21 days has a lot of certainty. But it still remains a 20-day estimate for what is inevitably a significant piece of work. But an estimate of between 20 days and 40 days conveys a good deal of uncertainty, as well as conveying an item of significant size.

When Story Points and T-Shirt Sizes Are Still Useful

At the outset of a project (or in the face of any new knowledge and emerging understanding) when there is very little clarity in what is to be built, story points and t-shirts can be a useful placeholder for more precise estimation.

When there isn’t yet clarity of what is to be built, time range estimates can convey a level of specificity that falsely conveys more understanding than we actually have. We should never communicate with more specificity or certainty than we’re in possession of. Story point and t-shirt size estimates are a useful tool to keep us from implying that we’re in command of things that are still just out of reach.

We can fall back to loose estimates in story points or t-shirt sizes when we lack clarity, but it’s understood that these placeholder estimates will be replaced with more specific time range estimate when we get the clarity that we need.

More Formal Techniques

There are a handful of more formal techniques based on range estimates.

For example, you can derive probabilities from the ranges. Or you can add a best-guess estimate to the optimistic/pessimistic range and triangulate and confidence intervals.

Using statistical methods to derive probabilities and confidence intervals can lead to the mistake of conveying more specificity and certainty than the underlying calculations can deliver. A detailed discussion of such approaches is beyond the scope of this article. There is an incredible amount of documentation available on such techniques readily available to anyone who wants to dig deeper.

You can go a long way with two-pointed optimistic/pessimistic before engaging any of the more esoteric extensions of the technique.

A Note on Agile Project Management Tools

It’s likely that your organization has adopted an Agile project management tool that has become indistinguishable from your team’s process. It’s also likely that your tool of choice doesn’t have provisions for anything but single-pointed story point and t-shirt size estimates, and doesn’t allow you to record range estimates.

The limitations of your tools should never be allowed to limit the improvement and advancement of the process itself. If your tool doesn’t allow the recording of range estimates, then don’t use that tool for estimates.

It’s important to note that your organization, at some point in its recent history, made the transition to the tooling you’re using now. In the mid 2000s, we all made transitions to Agile tools. We have all already demonstrated an ability to make such transitions. Not to downplay the effort, but transitions to new tools is far from insurmountable.

As a concrete suggestion, you can easily keep work item range estimates in a Google Spreadsheet that your team can share. You might balk at the notion of using multiple tools, but such is a characteristic of transitions. Having the estimate data in a spreadsheet also allow you to interactively explore what-if scenarios and experiment with some of the probabilistic techniques.

Tools always lag behind techniques. Which, by the way, is why Agile pioneers and leaders have always cautioned against allowing any tool to become so entagled with a team’s development process that it halts or complicates its improvement process.

Conclusion

Point estimation was a specific countermeasure for a specific set of conditions, conditions that largely aren’t as much of a problem anymore. It’s not that point estimates aren’t useful, but their utility is confined to circumstances and conditions that are increasingly rare 20 years into Agile.

Adopting time range estimates not only frees you from rituals and ceremonies that were more useful 20 years ago, but it will also bring you into greater alignment with the rest of the business you operate within. And doing that will increase the business’s trust in you and increase your credibility.

Trust and credibility are assets that you can leverage toward making even more advancements and improvements going forward.

SAGE: Early Warning System for Design Mistakes

2019-10-12T00:00:00+00:00

SAGE is a design quality assurance technique that can keep your work from becoming more difficult and complicated than it should have been. Looking at your software through the lens of SAGE quadrants can keep you safe from the most common setbacks and slow downs in software projects.

Design mistakes are subtle, and not immediately visible. Once a design mistake has been hardened and entrenched by afference, it’s often impossible to correct. SAGE makes design mistakes visible while it’s not yet too late to fix them.

What is SAGE

SAGE is an acronym that stands for Specialization, Afference, Generalization, and Efference. Each of these four are measurable software design qualities.

The four SAGE qualities can be visualized as quadrants, and used as a way to determine whether your design will be a magnet for trouble.

Afference and efference are opposites of each other, and generalization and specialization are opposite of each other.

Afference and efference form the coupling dimension, and generalization and specialization form the conceptual dimension

There are two quadrants that are harmful and two quadrants that are safe. If your code finds its way into the harmful quadrants, your productivity will start to suffer. The longer you leave these problems unaddressed, the more they radiate outward through the lines of coupling to adjacent parts of the implementation.

The happy quadrants have qualities that enable us to work without unintended consequences and unnecessary complications.

The unhappy quadrants feature the counter-productivity that radiates outward, expands, and compounds exponentially.

An Example: Model, View, Controller

Controllers are close to the user interface. So, they’re right next to the user. They represent specific things that a user wants to get done.

They’re the scenarios. They’re use cases. They correlate to features like “log in”, and “post a message”, and “make payment”. Controllers are the specialized parts.

Models are general. They’re used by numerous other classes, often controllers. Those other classes may have diverse goals and responsibilities, but they all use the model in the exact same way. A model is generally-applicable to a number of scenarios. Those scenarios have far more diverse and variant behaviors than the model does.

Nothing really makes calls to a controller except for the app server or web app framework infrastructure. There’s a front controller or router sitting in front of a controller that drives the controller. But there isn’t really much in the way of afference, and the afference that does exist is highly standardized.

Controllers are efferent. They call out to and coordinate other parts of the system, including models.

Models receive calls from potentially many controllers or other parts of the system. A model may have ten different controllers calling it. The model is general-purpose. It can be used in diverse scenarios. A controller represents only one of those diverse scenarios, and serves that single scenario.

The general-purpose code goes into the model: the model is general. The model is used by lots of other things: the model is afferent.

A good SAGE assessment of a healthy implementation built on MVC patterns will have the right things in the right places:

The General/Special Rule: Everything in Its Place

The underlying design rule is straight-forward: Put general-purpose things in general-purpose places, and put special-purpose things in special-purpose places.

Never get this mixed up and never make exceptions. If you find yourself between a rock and a hard place and tempted to make an exception then there’s a problem elsewhere with the design. Find it and fix it. Don’t let it fester in your code. You’re already seeing the secondary effects of viral infection due to adjacency.

If, borrowing from the MVC example, you end up with scenario-specific controller logic in general-purpose model code, that reduces the cohesion of the model, which in turn attracts parasitic afference to that model, which will make it difficult-to-impossible to improve or to correct. Where ever you put afference attractors, make sure that you’re certain of that design. You won’t get very many opportunities to make corrections if you get it wrong.

Coupling Axis: Afferent to Efferent

Afferent coupling is inbound or inward coupling. Efferent coupling is outward coupling.

Afferent coupling is like the walls closing in in the garbage compactor in Star Wars. The more that a piece of software is used or called by other pieces of software, the less liberty or license it has to be able change; either to be corrected for some defect or to be improved for some new realization. Every change means making changes to all the other parts of software. And those changes can have cascading effects along their lines of coupling.

Efferent coupling is the force exerted outward from some piece of software. A call made to another class is an outward expression from the perspective of the software making the call.

An afferent piece of software is the receiver of coupling. An efferent is the sender of coupling. The caller is efferent. The callee is afferent.

Some unit of software can be highly afferent or highly efferent or something in the middle with equal parts afference and efference. Those things in the middle ground might be naturally challenging to work on. Those challenges can be amplified beyond reach if coupling is out of alignment.

Conceptual Axis: General to Special

General and special apply to software in a couple of ways. There’s the generalization and specialization of abstraction, sub-classing, polymorphism, role interfaces, and such. And there’s the notion of something being general-purpose or special-purpose. Both of these dimensions are reflected by the conceptual axis.

A special-purpose piece of software carries out some particular usage scenario. It typically coordinates an assortment of other objects or things to get its job done.

General-purpose things are often also generalizations, aka: abstractions. Special-purpose things are often also concrete, like subclasses of some abstract adapter.

Implications

Things that are highly afferent and general are natural and more resilient to change. Things that are highly efferent and specific are also natural and more resilient to change.

Building upon design mistakes that are afferent will project the mistakes into the software that is adjacent to the mistake. And because software is not limited by physical space, a design mistake can infect an infinite number of other pieces of software that makes calls to the flawed pieces.

When the right coupling is in the right place, the adjacent software improves sympathetically, and doesn’t radiate sickness outward to other parts of the implementation. It stays healthy and easy to work with.

If you mix up the general and the special, and put special-purpose code in a general-purpose place in the design, or put general-purpose code in a special-purpose place, the result will be very unnatural coupling and design flaws that cascade out along those unnatural lines of coupling.

The same can be said for putting general and abstract code in a specialized, concrete place, and putting specialized code in a generalization.

Happy Quadrant: Afferent/General

This quadrant features inbound calls that limit the receiver’s ability to change, and either general-purpose or abstract, generalized code that should not need to change.

A highly-afferent piece of software is difficult to change. The pieces of your software that are highly-afferent should not be expected to change. These pieces are stable. Afference has less impact when it is on pieces of software that are stable.

Happy Quadrant: Efferent/Special

Efferent pieces of software are typically coordinators of other pieces of software. They make more outbound calls to other things that they coordinate than they receive calls from other things.

Because they make more calls than they receive, they’re easier to change. They have low afference.

Unhappy Quadrant: Afferent/Special

This quadrant features inbound calls that limit the ability to make a change, but also features scenario-specific logic which will likely cause the code in this quadrant to need to change.

Code in this quadrant is usually scenario-specific code. Scenario code is closer to particular user, usage, and business requirements. These things tend to be more capricious. They tend to need to change more often.

Capricious things are inherently unstable. Afference shouldn’t be combined with instability.

Unhappy Quadrant: Efferent/General

A piece of software that is both general and efferent, would make calls out to other pieces of software, but like a controller, would not receive calls from other pieces of software.

However, it would also be general-purpose or abstract. It would be coordinating a number of other pieces of software, and it would be a stable, unchanging implementation of such coordination of other things.

The absence of afference on a general-purpose, multi-use piece of software is a bit of a contradiction. It’s certainly possible to have parts of your design that fit this quadrant, but they’re more uncommon and somewhat odd.

Conclusion

SAGE can help you immediately spot the real trouble spots and fix them immediately. And the more you use it and apply it, the more automatic and instinctual it will be.

SAGE is a guideline. You can make hard measures of coupling and generality and abstractness, but you still need to be able to make a judgment call about what it means to your design. Sometimes your decisions will be straightforward, and sometimes it’s not going to be so clear, and you’ll need to invest a little more thought into it.

Understanding the interplay of coupling and generalization is absolutely critical to writing code that doesn’t drag you into its own hell. It’s also a good entry point into the ecosystem of design principles and design qualities that will only make your job easier and your work more rewarding.

Prior Art

SAGE is effectively a restatement of Bob Martin's Distance from the Main Sequence metric.

Afference: The Silent Killer

2019-10-05T12:00:00+00:00

Afferent coupling, or afference is the vice that holds your design in place and keeps it from changing. It’s like cement poured around your code and allowed to harden. It calcifies your codebase, keeping you from the productivity you know you should have.

You know in your bones that software development should be easier than it is, that it should be less tedious than it is, and should not be so fraught with setbacks and obstructions to progress. Much of this strife is rooted in afference. Especially unmanaged afference.

The elements of your codebase that have higher afference are the ones that you’re not free to change. Keeping afference (and really, all design principles and qualities) foremost in your mind while you’re writing code will keep you from adding new setbacks and frustrations to your work, and keep the existing ones from getting any worse.

What Is Afference?

Afference is the inbound coupling of some unit of software, be it a module, a class, a function, a library, a package, an interface, a system, or even a variable.

Inbound coupling means: uses of some unit.

In the following diagram, a unit (ie: class, object, function, module, library, package, etc) named Thing 2 is invoking some function/method/procedure on Thing 1.

That invocation that is inbound to Thing 1 is the afferent coupling.

From the perspective of Thing 1, that invocation is afference. Meanwhile, from the perspective of Thing 2, the invocation is efference.

Inbound coupling is afference. Outbound coupling is efference. Any invocation or call has both afference and efference.

The thing that is invoking a call is the efferent, and the thing receiving the call is the afferent.

A good way to remember the difference is to reflect on the meaning of the words affect and effect. You are affected by something that acts upon you. That thing doing the act is effecting the act.

The Problem With Afference

Software is largely useless unless some unit of software is invoking some other unit of software. Things calling other things are necessary and inevitable.

The problem arises from unmanaged afference, and from putting afference in the wrong place. Like cholesterol, there’s good afference and bad afference. Or more accurately, there’s safe afference and there’s dangerous afference. There’s afference that’s going to harm your ability to make progress with your codebase, and there’s afference that won’t.

When it comes time for you to make a needed change to that function on Thing 1, you may also need to make changes to all users of that function on Thing 1. Which means that changes to Thing 1 are likely to cause needed changes to Thing 2.

In the contrived example of Thing 1 and Thing 2, it might not be such a big deal. It depends on whether there are secondary effects to Thing 2. For example, if the call to Thing 1 requires that Thing 2 changes aren’t just limited to internals, but that Thing 2 requires changes to the arguments passed to it, then users of Thing 2 will also require changes. And so on and so on and so on.

This kind of cascade of changes is a sign of another, pre-existing design flaw where other basic design principles and qualities were neglected, and are being amplified and exacerbated by the addition of inappropriate afference to an already-problematic design.

This simple use case of one piece of software coupling to one other piece of software isn’t an entirely realistic representation of the reality facing most software developers.

This situation may not be much of a setback:

But such an overly-simplified situation is not the kind of pervasive problem that stops software developers and projects in their tracks. Typically, the kind of afference problem that software developers face looks like this:

The software sickness is intensified and aggravated when the obvious design mistakes aren’t recognized to begin with or are naively disregarded as “negligible”, and are further entrenched by continuing to build new software upon the mistakes rather than making corrections immediately when problems are identified.

Anything built upon a weak foundation is also weak. And in software, everything is potentially the foundation of the next thing that gets built.

In software, anything can be the foundation of anything else. Avoiding this state is one of the principle concerns of software design; it’s principles, qualities, processes, and patterns.

Implications of Afference

When developers come face-to-face with some work that they need to do on some piece of software with high afference, their motivation for the work wilts.

Software with high afference typically also suffers from afflictions that have accumulated much more intensely as a result of design flaws migrating along lines of communication and coupling.

By the time a developer comes along to make what would ordinarily be a straight-forward change, they’re stopped in their tracks in the face of entanglements between other pieces of software that interact with the one needing to be changed.

This is the cement that keeps software from changing. This is the calcification that keeps developers from being able to do their jobs in as straight-forward a manner as would be expected. And this is why software projects lose momentum over time rather than maintain or accelerate their pace of progress. And subsequently, this is why there’s so much staff turnover in software development.

Developers simply arrive at a point of maximum despondency with the codebases they they’re tasked with, and they start to look for greener pastures. And tragically, when a developer finds a greener pasture, what do they often turn around and afflict this healthy software with the same old software cancer that they were trying to escape to begin with.

Afference is never so insidious as when afference on a flawed design is allowed to increase as new software is built upon mistakes. This is how software developers tie their own hands and lock themselves into entrenched design mistakes that are the source of their counter-productivity.

What You Can Do About It

The first step to bringing harmful afference under control is to simply practice being aware of afference in every moment of your work life as someone who is at risk of creating yet more of it. As a developer, it’s in your hands. Vigilance and awareness is the first step.

Just make it a Big Deal. And you should, because it is indeed a big deal. Put it foremost in your mind and then you’ll be reminded of it while you’re up to your elbows in code. If you’re a developer who is prone to forget about the things that you’re supposed to remember, consider making yourself a checklist, and keep it in sight on your workbench.

You must also gain a grasp of which kinds of afference are safe and which are dangerous. Not all afference leads to exacerbated counter-productivity. It’s only when some piece of software becomes highly afferent that shouldn’t be highly afferent that the problems start.

The obvious implication here is that some pieces of software can afford to be afferent, and that afference will have a minimal impact on productivity, and some pieces of software absolutely cannot afford to be allowed to become highly-afferent. It’s critical to put afference where it belongs and where it can do less harm. It has to be something that is entirely under your control as a developer.

Afference is safe for generalizations and general purpose things, but dangerous for specialized things and special purpose things. The underlying principle is that things that don’t change at all or change very infrequently can afford to have high afference, and things that change frequently are likely to be highly efferent, and therefore should not also be afferent.

For example, a scenario-specific thing like a web controller is a special-purpose thing. It’s only used when serving the specific use case that it implements. It is not expected for something like a web controller to be afferent. It coordinates and invokes other things that are enlisted to carry out some web request. It’s efferent. It’s relatively easy to change the code in a web controller because most of its communications are outbound. Most changes to a web controller are changes to internal logic.

On the other end of the spectrum, something like the a language’s ultimate base class, like Java’s System.Object class or Ruby’s Object class, are highly afferent. Absolutely everything in the language is coupled to these apex classes. They are highly afferent, and like things that are highly afferent, they are not expected to change - at least, not without some formality of process.

Conclusion

Afferent parts of software aren’t amenable to change, so parts that are expected to change should not be afferent.

Afference is unavoidable, but it’s manageable. And managing it is a critical component of keeping your project from turning into the tedious drudgery that we all want to avoid in our work lives.

It’s not the existence of afference that kills software development productivity, it’s putting afference in the wrong parts of the design: the parts that are expected to change.

Just by keeping an eye on afference, and conditioning yourself to be aware of it and where you’re putting it, you’re at least a step ahead of its worst effects.

Science Denial and Software Development

2019-09-19T12:00:00+00:00

Software development decisions are often not scientific. We make technical decisions, and we often mistake the technical for the scientific, but our decision-making process is usually one that often reflects ideological machinations and adherences rather than the objective assessment that requires us to question the things we’ve chosen to cling to.

Without an objective standard to determine whether our software decisions are either helpful or harmful, we’re left with the worst possible means of arriving at conclusions: We do what others are doing. And this is the principal means of decision making in most of software development.

We reduce to a mere popularity contest something that is already able to be distilled to objective measures. And because we’re relying on something as flimsy and subjective as a popularity contest, we volunteer ourselves to the unscrupulous manipulations that win popularity contests. In the process, we remove ourselves from the sphere of influence of the very knowledge and learning made available to software developers for assessing the soundness of our decisions and the software that results from them. And in the most heartbreaking of cases, we allow bullying and social shaming to have the last word on matters that are readily-decided by plain old structural dynamics and structural physics.

This isn’t something that’s unique to software development. It’s common to every emerging technical field, and software development’s scant 3 score years puts it squarely in the “emerging” category.

In their lifetimes, both Louis Pasteur (1822 - 1895) and Ignaz Semmelweis (1818 - 1865) pleaded with the overwhelming majority in the medical community to sanitize their hands and instruments or else risk contaminating and killing their patients. The germ theory of disease was largely unwelcomed by physicians of the age, despite having been proven by Agostino Bassi in experiments decades earlier.

Physicians could not detect the subtle factors causing sickness. Worse yet, physicians rejected the very possibility that they could be the very cause of sickness. They were, after all, the epitome of the scientific hero image of their time. How could such grand and anointed magicians possibly be wrong?

How could something that causes such a flagrant and obvious effect as the death of a physician’s patient be beyond the ability for such grand figures to detect?

And of course the answer is trivially-obvious: The causes of disease were beyond the unaided perceptions of any human being who has ever lived and who will ever live. But the physicians took the possibility of not being supermen personally. And they not only rejected the science outright, they attacked it with great vigor and violence, seeking to discredit it and destroy the reputations of its purveyors.

Software development is largely at the same point of evolution in its time as medicine was in its time in the 1800s. And like all contemporaneous adherences, we cannot accept that we the grand and glorious magicians of code might possibly be wrong - really, really wrong. After all, we represent the ultimate evolution of software development thinking - even though software, no matter how high tech, remains largely a human field that has not outgrown its adherence to superstition and idolatry.

And like those physicians of the 1800s, when we 21st-century software developers put our hands on our patients, their viability tends to get worse over time rather than better.

In 2006, a now-infamous idea was introduced to the world of ORM+MVC web development: Fat Model, Skinny Controller. Whatever the original intent or value of the advice, it became deeply entrenched and utterly pervasive throughout the culture. Especially in the Ruby on Rails world where it originated during Rails’ infancy.

The problem: It was objectively wrong. It was always wrong. It was knowably wrong from the very instant it was conceived, and remains wrong to this day. It’s wrong because it neglects and contradicts the very science that predicts whether adherence to this heuristic will cause all subsequent work to become far more complicated, far more error-prone, and far more time-consuming. It will always be wrong.

It neglects the effects of afferent coupling on general-purpose modules. By doing so, it invites low cohesion which in turn acts as a feedback loop amplifying the vicious cycle of high afference on high generality. And even as I say this, I can hear countless voices saying to themselves, “This is just academic navel-gazing and bike-shedding that won’t ever be as practical as the things I need to get done in my job today.” And yet most of the setbacks, strife, and lost productivity in the day-to-day work of contemporary ORM+MVC styled application development can be explained (and predicted) by these principles.

And yet, the Rails community accepted this bit of pseudo-science as unquestionable edict. And that’s all it was: edict. Dictated, but never substantiated against objective measures.

Had it been examined through the lens of software design principles, it would have been rejected on the spot. It would not have taken a decade of progress away from those who would cling to the dictate without question. For it was handed down to the mere plebeian caste by the keepers of the great and unfathomable knowledge that shall not be questioned lest we risk reprisals from community gatekeepers. And who are common developers to question the sonorous and mellifluous voices of the great code magicians of our time? For they are pleasant to listen to, and how could anything with the appearance of pleasantry possibly be mistaken?

Meanwhile, Louis Pasteur’s voice echoes hollow in the fading distance: “Wash your hands! Wash your hands!”

“Navel-gazing” and “bike-shedding”. These are the word-weapons of the anti-science advocate. They seek to silence inquiry and dialog. They insinuate harm to reputation and thus livelihood. They’re the audible calling card of the tyrant hell-bent on not being unmasked as an impostor, not just one with impostor syndrome.

These words have no place in software development. That is, at least not a software development this is rooted in science rather than cult fundamentalism. They have no place in an organization holding the forces of entropy at bay long enough to allow the roots of a learning culture to become viable and self-sufficient.

The software design body of knowledge - the design qualities, the design principles, the patterns that derive from them, and the processes that guide us through the dark - is the science of programming. It’s the objective arbiter of software development decisions. It loosens the grip of subjectivity on software development decisions and externalizes them so that they’re not based exclusively on the fog of emotional responses. It denies an opportunity for bullying and shaming to extort outcomes that benefit the ease and luxury of some colleagues at the cost of the continued strife of others.

The application of software design science still leaves room for interpretation and disagreements about interpretations, good faith negotiation of meaning amongst colleagues, and an ultimate decision maker who has both the authority to break a stalemate and the integrity to make the decision for the good of keeping subsequent work free from the complications and setbacks of short-term decision-making and local optima effects.

It’s critical for countless reasons for developers to start applying software development science to software development science work in every moment, in every act, in every conversation, and every line of code. When we do this, we begin the journey of elevating software development from the depths artisanal arts and crafts subject to the whims of personal preference and personality cult to an honored place in the pantheon of production and product development industry.

Software Cancer

2019-09-12T12:00:00+00:00

Software doesn’t go bad over night. Like cancer, a whole tumor doesn’t just materialize in the body in an instant. An entire system isn’t pervasively compromised in a single spasm.

Software cancer starts at a single cell. The time to deal with it is when it’s still a single cell. Otherwise, it will expand exponentially until it pervades every aspect of a system and can’t be controlled.

Even when software is suffused with cancerous tendrils pervading every aspect of a system, developers may still not recognize that it’s there.

We’ve become so complacent with unhealthy software design that we’ve entangled the very identity of software developer with the ability to tough it out and laugh it off. At our worst, we even demean and deride the very body of knowledge that is specifically constructed to allow us to avoid the cancer, and to detect it at its earliest stages.

Software design and its four pillars, namely design principles, design qualities (and their metrics), design patterns, and design process, are typically not understood well-enough at large to put them into moment-by-moment practice. Most developers have a passing familiarity with some of the names of some of the principles, for example, but don’t really have a full grasp of them, or how to use them as concrete and tangible tools.

It’s these tools that are the mental guardrails that protect our work from problems that we can’t afford. The challenge is that, continuing the analogy, software design defects are effectively invisible to the unaccustomed eye. They exist at a level the isn’t made readily-apparent by merely looking at the code, not unless the developer is already well-practiced in the software design body of knowledge and practice.

Software design is the science of programming. Design qualities, design patterns, and design process build upon each other, and together form the early detection system that identifies and then eradicates cancer while it is still within our power to control. It’s an instrument like a microscope that allows us to see things that the naked eye might ordinarily disregard while the mind is fixed on more tangible and concrete aspects of getting the software to just work at all.

The science of software design teaches us which things we do in software development are mistakes. And it also informs us why a take-no-prisoners stance toward design mistakes is critical to maintaining healthy software, and why even a single cancer cell must be dealt with.

But software design isn’t microscopic. Software design is subtle. And because it’s subtle, it’s a lot easier to miss the mistakes.

It’s the mistakes that are the cancer. Design mistakes may have no effect on the operation of a software system or software application’s features. Design mistakes don’t always result in malfunctions of the software. Design mistakes result in malfunctions of the software team and the software project.

The subtle mistakes in this subtle realm compound upon each other exponentially until software gets to the point where we can no longer make progress with it at a pace that we’d want, and that business leaders expect.

And we laugh it off. We say it’s an inevitable outcome of software, and that it’s part of software’s inherent nature. And worse, we disregard the fact that our ability to make progress continues to decline as we continue to put our hands on our code.

We reject the very possibility that microscopic influences could have any effect on our codebases irrespective of the fact that our codebases are assembled from countless numbers of them.

But software codebases and software projects don’t just go from being healthy one day to being riddled with terminal cancer the next. That cycle of decline and decay isn’t part of software’s nature at all.

There isn’t a single atom or iota of disease that finds its way into the body of software that isn’t put there by developers. It might be done by negligence or by ignorance or even by cynical science denial, but this is the inescapable nature and reality of software.

There are no evil elves that sneak into our repos in the middle of the night and wreck havoc on the good code we left there at the end of the day. If code goes bad, it’s because we set that decay into motion. If the implementation gets better, that’s to our credit. If it gets worse, that responsibility rests only with us software developers. If we proceed to do nothing about it and even disregard the very body of knowledge that we’re supposed to be armed with in order to save of such effects, that’s on our shoulders.

There isn’t a troubled software project in existence whose troubles aren’t complicated and multiplied by design mistakes. There isn’t a team that’s struggling with lower-than-expected productivity that hasn’t disregarded the design science of programming to some degree believing that it’s an innocent gambit.

The best cure for software cancer not surprisingly is prevention. And if not prevention, early detection. Cynicism toward software design and brute force heroism and bravado are just unhealthy coping mechanisms that engender more calcification and the need to justify yet more science denial.

The cure is to put software design, it’s qualities, principles, patterns, and processes front and center in every decision about every line of code, and to leverage software design as a purposeful act engaged with our full attention.

Happy Birthday, Nathan!

2019-09-12T12:00:00+00:00

Join me in wishing Nathan a happy 1st birthday. The first birthday after escaping death by grace of some inconceivably small margin.

He fell 45 feet and spent the next six months injured so completely that “smashed” was the word that kept coming to mind when people would ask for updates.

Smashed feet first, so they were the first things to break. The shock wave initiating at the metatarsus, into the tarsus and firing straight back. A microsecond later later the heels explode in a crossfire distributor beam of tarsal detonation and heel impact. In the next instant, the pressure wave and the inbound skeleton smash through both shins. The detonation wave jogs to the left and smashes through the femur, jumps the pelvis, and completely spares the rib cage. Nah, just kidding. It smashes through eight ribs on one side, seven on the other, and breaks the sternum as it exits the chest and heads for the head. Neck and spine snap forward, rupturing the jugular vein. If you look at Nathan today, it’s not exactly the same face that smashes into the turf a microsecond later. An incision from ear to ear, over the top of the head is how the 3D-printed replacement cheekbones are installed. The jaw is smashed into literally uncountable fractures and is wired shut for months. The nose smashed inward, remaining a jagged break, and inward, concave scoop to this day. A tracheotomy tube goes into that divot in the throat and a feeding tube through the abdominal wall. The worst of the leg injuries are plated. Several surgeries are smashed together in the first night. I lost count of how many surgeries followed.

Scene: End Day 1.

Months in, when Nathan was finally allowed take in sustenance in a way that didn’t involve having it pumped through a tube directly into his stomach to eat, the hospital happened to be serving fish tacos that day. So they tossed the taco’s tortilla, fish and all, into a blender and served it as a disgusting, liquefied stink smoothie. I wouldn’t be surprised at this point if the story proceeded with, “And then they poured itching powder into his bandages.” The entire thing is one unending, indignant kick in the nuts.

While we’re on the subject, the nuts weren’t smashed. Neither were any organs. The spine wasn’t smashed. The eyes weren’t smashed. No brain injuries and no spinal injuries. No nerve damage anywhere along the shock wave’s path of destruction through his body. Some teeth were smashed. An inconceivable stroke of luck, as far as you can use “luck” to characterize any of this.

The tiniest fraction of infinitesimal luck. Just enough so that we’re not having conversations about Nathan’s legacy today. But let’s talk legacy.

One of the most insightful, self-examining, honorable people I’ve ever worked with in software, and one of the best friends a person can have. Steadfastly impervious to the geek psychological traps that draw software people to the cheap thrills of joining in on a mob attack like moths to the fire. Truly committed to integrity. Truly committed to diversity. Truly driven by courage. Committed to leaving things better than it was found, and leaving no trace of self-aggrandizement. More courage then the next thousand geeks combined. Disposed of a deep knowledge of software design and software process, immune to the fads that blatantly neglect fundamental principles, and unyielding to the pressure to abandon principles. And capable of wielding software development from the embedded system up to the web UI. And of course, the Eventide project would still be largely hypothetical if he hadn’t written the hosting infrastructure that actually lets it run on an actual computer, and built the only testing framework I’ve ever seen that is positively joy incarnate.

So, on Nathan’s birthday, the well-deserved aggrandizement can be our charge. Not just for amazing work in software and software community; in being the change you’d want to see in the world, but for surviving against staggering odds, and remaining dignified in the face innumerable and unrelenting indignities.

Happy Birthday, Nathan. This year has kicked you relentlessly while you were down, and while you can’t quite kick back yet, I’m glad that today we can at least get burgers now without the existential insult of having to run them through the food processor. And hopefully on your birthday next year you’ll be walking again, the rest of the surgeries will be in the rear view mirror, and we’ll all be somewhere raising a glass to the fact that we can toast incredibly near misses rather than absent friends. No one has to tell you to stay strong. I’ve never seen someone put up with so much all at once. You make my life better, and I can barely bring myself to imagine the alternative.

And I know you don’t want me to disclose that this is your favorite song, and how deeply it moves you when it comes on, but on the occasion of your first birthday since the accident and your incredible survival, I think people deserve to know full depth of your heart. Especially because you’d never think of seeking glorification for yourself, your work, and your life. You can seek reprisal later, but for now I just wish they could see this side of you the way I have been able to, and be inspired by your spirit:

🤣

Subtle Knowledge and the Limits of Knowledge Transfer

2018-06-13T12:00:00+00:00

One of the great frustrations with acquiring subtle knowledge is that it can’t be materially transferred.

The one thing that definitively characterizes crude knowledge is its ability to be rendered into books, blog posts, video materials, and the ability for anyone who consumes those materials to gain a certain measure of ability or fluency with the subject.

Subtle knowledge can’t be assimilated through material means. It can’t be converted into concrete materials the way that more basic and crude subjects can. The names of more subtle notions and their descriptions can certainly be written down, and they can even be demonstrated to an extent, but by no means does the presence of such artifacts allow someone else to be instantly augmented.

You can be told the names of subtle things, like design principles and design qualities, and you may even come to be able to recite their definitions from memory. But that is only memory. It is just intellectual inventory. It’s not necessarily actionable. It’s not something that you can put to use on-demand. Not until you’ve been reflecting on it for a while within the context of the particular circumstances your own hands-on work.

An intellectual inventory is a necessary precursor to having working knowledge of the subtle, but an intellectual inventory on its own only allows you to imitate and mimic the most externalizable and observable aspects of the work.

Being constantly distracted by the content of the intellectual inventory does little more than create an intellectual materialism, where valuing yourself and your abilities rests squarely on the volume of things you’ve memorized and put into your inventory without consideration of making them actionable. It creates a propensity to value the shallow acquisition of many new items of the inventory as quickly as possible, rather than gaining much transformative understanding.

Subtle knowledge is transformative. Crude knowledge is just cumulative. It’s necessary, but it’s only the very beginning. Subtle knowledge brings breakthroughs in understanding that changes the way you use all the more crude things you’ve learned. It changes how you do your work, and how you employ those crude, material items items of your intellectual inventory. Crude knowledge is those things that can be memorized from consuming the artifacts of someone else’s work. Subtle knowledge requires your willing participation to transform it into something as readily-actionable as crude knowledge.

At the crude end of the spectrum, you can get by on repeating and mimicking idioms and conventions. But it’s the subtle–the principles and the qualities that principles are built on-that tell you when the application of some crude thing is going to be the mistake that you only learn to regret in the future.

And yet, this incredibly powerful knowledge can’t be transfered from person-to-person by rendering it into materials. Only the crude can be converted to material. Only the basics can be transfered that way.

Gaining fluency with the subtle end of the spectrum takes practice and reflection. And this necessity is what rubs some developers the wrong way, turns them off the whole idea of going beyond the crude, and even makes them angsty and angry. Sometimes even to the point of denying the very possibility of the existence of the science of software design.

As we sharpen up our design senses, things that seemed to be imperceptible start to become obvious. Things that were distant and esoteric become close-at-hand to our work and readily-actionable.

It’s only because we haven’t been introduced to the more powerful aspects of software development that we tend to not actively pursue them as a matter of course for skills and career development. They’re imperceptible at first. They seem tiny and negligible. But as our senses become more acclimated to the subtle, the subtle things become as large and obvious as an elephant.

It’s the intentional improvement our ocular instruments that allows us to see subtleties as painfully-obvious things. And at that point, they’re not so subtle anymore. It’s only in retrospect that we still see them as subtle.

And even if you don’t understand the subtle and powerful side of the software development capabilities continuum enough to put them into action in your day-to-day work, you can still practice them merely by becoming aware of what they are, and paying attention to them. You can do this even if you can’t do anything about the problems with the software you work on. In fact, it’s a precondition for ever being able to have these subtle things available to your thinking in every moment, keeping you from mistakes, and transforming your more crude knowledge into something that doesn’t suffer from the pathological naiveté of intellectual materialism.

You need the crude. After all, you can’t built software without languages and tools and frameworks and patterns. But the subtle is the key in the lock that switches these crude things on and brings them to life and makes them powerful.

But you’re simply not going to absorb subtle knowledge by tucking a kindle under your pillow at night and hoping for knowledge to just leak into your being. Not are you going to acquire it through passive reception like reading or watching demos.

Gaining subtle knowledge is always a process of breakthroughs. The most powerful knowledge is the most subtle. And the most subtle is only ever realized through the kind of participation that allows for breakthroughs.

It won’t happen over night. But it won’t happen ever if subtle knowledge never gets the benefit of your attention.

Design Is Subtle Knowledge

2018-06-12T12:00:00+00:00

There’s a spectrum of things you can hold in your head, from crude knowledge to subtle knowledge. Things like tools, frameworks, and patterns are on the crude side of the spectrum. Design principles and design qualities are on the subtle side.

Knowledge of design qualities and design principles is much harder to acquire than the more crude subjects, but that subtle knowledge is a multiplier of all the other software knowledge you have.

All of the design principles you know or have heard of are expressions of a handful of qualities. Some of the more critical qualities to have a grasp of are afference and efference, generalization and specialization, and push and pull.

For example, the Tell, Don’t Ask principle can’t be expressed directly in terms of Push, Don’t Pull, which has a more common name: encapsulation. And each one of these qualities reflects cohesion and coupling. And furthermore, cohesion and coupling are inter-related and affect each other. Afference and efference are kinds of coupling. Afference is inbound coupling, and efference is outbound coupling.

All of these qualities direct how or when a particular tool or framework or pattern should be used. They keep you from mistakes like using the wrong tool, or using a tool in a way that violates design principles.

Without belaboring the details of what each one of these qualities means and the specifics of their impacts on design, it should be said that protecting developer productivity is the whole point of focus on design principles and design qualities.

It all boils down to adverse coupling. Coupling is the quality that dictates whether a team can move through its work without hindrance, or whether it will be bogged down.

It should also be said that adverse coupling is not a naturally-occurring phenomenon. Software doesn’t rot like a physical material. If it exists, it’s because developers put it there. And it’s unlikely that developers would willfully sabotage themselves and their own work. Harmful coupling exists in systems because developers routinely massively underestimate its effects.

The impacts of coupling and the means to control it is a subtle subject. It takes a lot longer to bring its meaning into focus than grasping a new tool, framework, or pattern. It doesn’t take forever, but it’s not instantaneous.

Securing the productivity of a software project requires protecting the ease of change. The interplay between software qualities (as expressed by design principles, and the patterns built on them, and the frameworks built on the patterns) predicts whether a team will be hindered in making necessary changes to the implementation, or whether they will sail right through. Having a grasp of these subtleties, and keeping them in-mind in every moment, keeps software projects from becoming the grinding slog that software projects often become.

It’s the subtle knowledge that keeps us from making predictable mistakes. It doesn’t so much help us to remediate past mistakes, because that cost is already sunk. The best we can do is sharpen up our design senses so that we don’t invite these mistakes; to sharpen our sense of the subtle so that it is no longer out-of-reach.

Design is subtle knowledge. It helps us to recognize the well-known, but subtle mistakes before they become calcified beneath weeks, months, or years of further development, and keeps the mistakes from amplifying each other.

Knowledge of design qualities and design principles is the most powerful knowledge you can have. It activates and amplifies your knowledge of the more crude and material things–the tools, frameworks, patterns, and even languages. It doesn’t obviate the need for more crude knowledge, but it does complete that crude knowledge and allow you to wield it in much more precise and deterministic ways.

Subtle knowledge makes it clear how you should use tools. It helps you avoid the traps that otherwise might suck you in — especially those that are merely prominent or fashionable, and that you’ll regret in the future as the subtleties are slowly revealed to you through series of unfortunate mistakes.

And while subtle knowledge isn’t as readily available as the crude subjects that can be materially transferred, merely by cultivating an interest in this body of knowledge and paying attention to it, you’ll start the process of opening up the other side of software development knowledge that makes everything you do more effective.

Subtle Knowledge. Crude Knowledge.

2018-06-11T12:00:00+00:00

Every mistake that you’ve made in software development was already a well-known mistake before you made it. Every mistake you’ve made in the past that continues impeding your ability to make progress at the pace you’d expected was known at the time you made it.

Every mistake you’re making right now is already a well-known mistake. You just have to learn to sharpen up your mental lens to see these mistakes before you make them, and avoid them.

You don’t have to personally experience every class of mistake to finally become attuned to them. You can know them merely by their fingerprints. You can know them through the fundamental design principles and design qualities that define software development.

But principles and qualities are subtle things. They’re subtle knowledge. You have to have a more fine-tuned sense of software to see them. But that doesn’t mean you have to wait years or decades to get that fine tuned sense. You can just acclimate yourself to them on purpose. And that will kick start the emergence of a finer sensibility of the subtle end of the knowledge spectrum.

Subtle knowledge — or knowledge of subtle things — is the most powerful knowledge you can be equipped with. It means the difference between making mistakes that you’ll regret, and that compound the complications of the other mistakes that you regret, and always moving forward without having to deal constantly with the setbacks that issue from those mistakes.

The challenge is that subtle knowledge isn’t easily-accessible through a lens that works at the magnification level where mostly the crude knowledge is found. And because of this, it’s common to reject the effects of subtle knowledge on the work. And it’s possible from this perspective to dismiss this power reflexively because it isn’t yet within view or within reach.

To the unpracticed eye, the subtleties are invisible. But they remain the foundation of everything. A cancer cell is invisible to the naked eye, but it’s impacts are undeniable. So too are the particles that act as countermeasures to it. When you’re working at the cellular or atomic level, you can have profound effect on the level of the organ or the organism. And the most pervasive, all-encompassing impact you can have on your software work are at that level.

Nonetheless, no amount of rejection of the very existence of subtle knowledge can obviate its object existence and its impacts. Knowledge of the subtle is power. It’s at that level of fine-tuning and super-fine tuning that the cancer that characterizes so much software is avoided. Developers don’t make mistakes because they want to. Developers make mistakes because their design sensing devices aren’t yet tuned to the most powerful level of conceiving of design.

Everyone starts at the beginning. And the beginning is the crude. At the beginning, just getting some code to work is a victory. Over time, it just fades into the background and becomes a given as we acquire an ever greater inventory of tools, and more experience with them.

Tools are on the crude end of the spectrum. Sooner or later we learn a thing or two about patterns. Patterns are a bit more subtle than tools, but patterns as well are crude.

Patterns and tools don’t tell us when we should use them and when to avoid using them. It’s design principles and design qualities that give us these signals.

Principles and qualities are on the other side of the spectrum: they’re subtle. In the beginning, while enjoying the victory of getting software to work at all, we may pay no attention to principles and qualities. Nonetheless, they’re still in-effect, and they’re still dictating the terms of whether we’ll regret how we’re using those more crude things.

Every developer can code. Every developer can use tools. Not every developer can see mistakes before they’re made. But every developer can become acclimated to the subtle knowledge that illuminates mistakes before they’re made–even if these things are presently invisible in the present. Design principles are subtle. Design qualities are even more subtle. Yet they are the most powerful things you can know and leverage. And they’re the difference between the enthusiasm you feel at the beginning of some work, and the enduring regret you have later.

The subtle knowledge is the ever present early warning system that keeps you from having to experience your work through the lens of eventual regret. You just have to practice seeing it, and making it a tangible part of every line of code you conceive of.