As engineers, we often face problems that are technical in nature. But we occasionally have to also deal with issues that come up in our interactions with others.

This post is about one such incident I experienced at a previous job, and what I learned from it. For the sake of privacy, I have changed the names of the people and entities involved.

Throw it Out

We should not be sinking any more time into a system that is going away.

That was the only comment that William, our most senior engineer, posted on my pull request.

I was stunned. How could I be so foolish? Why did I rush into writing code instead of realizing it would be a waste of time? And what would our CEO, Alex, think?

The Beginning

When I joined the startup First Depository, William had already been there for a year. After a few months of working with him and receiving his feedback, I came to accept that he knew what was best.

On top of that, our company had an explicit policy of hiring people who were “smart and nice”. Of course I could trust William.

I got accustomed to accepting his feedback and incorporating his requested changes into my pull requests without questioning him much.

Apostrophe

Fast forward two years, and I was dealing with an issue in our payment processing job. Because of the high volume of payments we were handling, the job experienced errors frequently.

That job, along with all of our other jobs, ran on Apostrophe, a jobs framework built by our co-founder and CEO, Alex, years earlier.

Apostrophe allowed for global settings. Those settings specified three retries for all jobs. But even with three retries, the payment processing job would still fail to complete at times.

These failures occurred often–sometimes multiple times a week–and required an engineer to quickly investigate the impact and occasionally remedy the situation.

I was modifying Apostrophe to allow custom settings per job. This would enable us to allow more retries of our payment processing job, decreasing the chances of a terminal failure.

After noticing that I was working on this change, Alex messaged me in Slack asking if he could take a crack at implementing the feature. Of course he could, but I continued working on my solution as well.

Once I got a mostly-working solution together, I submitted a pull request. Alex, who had been too busy that day to complete his implementation, commented that my changes looked pretty good. I had just missed one thing. So I made the correction and pushed up a commit.

I was expecting an approval from Alex at any moment when I saw the comment from William, suggesting I throw away my changes. After mulling over it for a day, I closed my pull request. I figured he was right again.

The Steps Project

Shortly after that, I had written a design document for how we would go about breaking up the payment processing job into smaller, discrete steps. Each step would complete when a record was written to the database.

By breaking up the job this way, each step would check where a record left off and pick it back up in the case of an error. Additionally, with more frequent commits, each database transaction would stay open for a shorter duration, reducing the chances of a serialization error on commit.

I chose to use virtual statuses, whereby our job would look at the status field in addition to several other fields in order to determine on which step the job left off.

I preferred virtual statuses over introducing additional status values to the database for each step, as the latter approach would require data cleanup in case we needed to roll back code changes. Additionally, introducing new statuses would require that we write even more changes to the database, which could result in more errors.

I listed a total of four or five bullet points containing my reasons for preferring virtual statuses and explained them in more detail over a few paragraphs that followed.

William’s response:

I think I disagree with all of your points, but I’m not sure I fully understand. Can you clarify?

I was once again stunned. Was it possible for someone to disagree with a position while not fully understanding it? Why did I seem to be wrong about everything lately?

Can I Please Speak to the Manager?

I brought up William’s resistance to my recent pull requests with my manager, Ryan, in our 1:1 meetings a couple of times. He didn’t offer much support other than suggesting that I push back more on William’s feedback.

Ryan’s boss, Bill, had specifically expressed to me that he wanted William’s eyes on the project, as he was very knowledgeable about that part of our codebase. So kicking him off of the project was not an option.

I realized that I would have to come up with a solution on my own.

Working Agreement

After thinking about it for quite some time, I realized I needed an overarching solution; one that would neutralize William’s unreasonable resistance to all of my work rather than addressing it in one particular project.

I came up with the idea of developing a Working Agreement: a document that would describe how all members of Engineering would work effectively with one another.

I wrote down a list of scenarios, including the ones I was facing with William, although generalized to not call him out by name. I planned to bring the list to the Engineering team, where it would serve as the starting point for a series of discussions in which we would shape the Working Agreement together.

I shared the list with Ryan. He liked it and recommended I present it at the next Brown Bag session.

Missed Opportunity

A couple of Brown Bags came and went, yet I couldn’t work up the nerve to present the document.

How could I present the document without calling William out in an unfair way? Was the act of presenting the document equivalent to me dictating how all the engineers should work? Was I wrong about the whole thing?

Before I got a chance to present the document, First Depository had a large reduction in force, which impacted half of the company, including William and me.

I never got to present it.

Seeing Things Clearly

After the layoff, I had to quickly look for a new job. I didn’t have time to deal with my feelings about William’s actions or having been laid off for the first time in my career.

Grief came and went, and once I settled down, I had a better understanding of what had happened.

William

When William shut down my Apostrophe pull request, he argued that spending time on a deprecated system was a waste of time.

A simple calculation could have settled it. Was the time I spent on the change less than the time it would have saved us having to deal with payment processing errors?

Without performing this calculation, holding a strong position is not possible. I spent a day on the changes, which is not a lot of time. There is a high chance that we would have seen a return on that investment in six months, let alone in the few years it would have likely taken us to migrate all jobs off of Apostrophe.

Yet, William didn’t provide this calculation with his argument. He only provided his disagreement.

William had a project in mind to migrate us off of Apostrophe, which he wanted to lead and I suspect he felt that remediating existing issues with our payment processing job in the short term weakened the case for him to work on that bigger project.

As for William’s resistance to my virtual statuses approach in the steps project, it’s not possible to disagree with a position if you don’t understand it.

You either understand an argument and can hold a position (agree or disagree), or you don’t understand it, in which case you should withhold judgment until you do understand it. And in order to understand an argument that was already explained, you should ask specific questions about where your confusion lies rather than just saying you don’t understand.

William was certainly smart enough to understand my explanation. He decided up front that he wanted to disagree, and by asking me to explain the whole thing again, he was keeping me busy while he looked for ways in which he could try to poke holes in my argument. And by not providing the reasoning behind his position, he was protecting his own argument from having holes poked in it in return.

William was arguing in bad faith: appearing to sincerely seek the truth but secretly prioritizing self-interest.

He wanted to work toward a promotion, and by appearing to be right and preventing those around him from shipping work on significant projects, he hoped to look better by comparison.

I also think William didn’t like when someone else arrived at the right solution first.

I experienced a situation with him a year prior to the Apostrophe project, where we had opposing views and a drawn-out discussion about a particular point. After I relented and went with his approach, he changed his mind to my original position.

He wanted to be right while someone else was wrong.

Ryan

As for my boss’s suggestion that I push back more, he wasn’t wrong. I should have pushed back more. But that didn’t mean that what William was doing was right.

There was plenty of work to do. William could have worked on something else instead of preventing me from doing my job.

Just saying “push back more” creates a culture where everyone assumes that their own solution is correct, and they fight to prove that rather than working together to figure out what the correct solution is.

William was pushing back too much, without trying to validate his opinion against facts and logic. And the burden of having to back up one’s opinions with facts and logic applies equally to everyone.

Ryan was likely too timid to talk to William or William’s boss about the situation. It was easier to tell me to deal with it myself. I also suspect that he didn’t want to bring it up with his own boss, Bill, as he would have appeared to not have things under control.

What Could I Have Done Differently?

Now for the important part. What could I have done differently?

I should have gone ahead with presenting the Working Agreement idea to the team while I had the chance, even at the risk of making William uncomfortable. This would have gotten me support from all of Engineering against William’s resistance to my work.

I could have also responded directly to William’s comments in my pull requests, asking him to provide facts and logic to support his positions. This would have gotten more eyes, including Alex’s, on the specific issues.

I was playing only defense, but I should have been playing some offense as well.

When I talked to Ryan, I should have made it clear that, as my manager, I needed his support and that if I didn’t get it, I would seek it from his boss, Bill. This would have put pressure on Ryan to take action for fear of his boss having to get involved.

Regardless of what the outcome would have been of the actions I should have taken, William had already exposed himself as being dishonest.

And because of that, I could never trust anything he said ever again.

If you think that sounds extreme, consider this:

If someone gave you a bottle of vitamin pills and it turned out that even one of those pills could be poisonous, would you take any of the pills in that bottle?

I wouldn’t.

Disagreeing for the Fun of It

Some time after getting laid off from First Depository, I found a video by Adam Savage that gave me some additional perspective on what I had experienced.

Adam was the co-host of the show MythBusters. In the video, he responds to a question from a fan who is starting a business with a friend.

While responding to the question, Adam relates how his co-host, Jamie, would often disagree with him without providing any reasoning, and enjoyed doing so.

Adam eventually learned to not let his co-host’s contrarian behavior bother him.

But why stop at ignoring it? If someone is intentionally being disruptive on the job, shouldn’t there be consequences?

I think Jamie should have had to paint the netting white.