Lean vs Agile – Interview with Sam McAfee

Lean Vs Agile

(Sam McAfee is the Chief Technology Officer of POPVOX and author of Startup Patterns.  I reached out to chat about his approach on Lean, startups and teamwork. You can find him on Twitter or Linkedin.)

Q: What are the differences between Agile Software Development and Lean Software Development?

A: Lean vs Agile: At the highest level, the difference between Agile and Lean software development is mainly one of focus. They are framed around different overall goals; in Agile’s case, it’s the quality of the software that is output by the engineering team, and in Lean’s case, it’s the value delivered to the customer by the entire organization. Of course, these focuses are complementary in the context of running a software business, but their specific implementations can sometimes leave them at odds with one another unless you’re very conscious of both in how you set up your production process.

The best way to understand the “lean vs agile” is to look briefly at the literature from which they each sprung. And let’s set aside “Lean Startup” for a moment as something separate, as we’ll come back to it in a bit.

Agile with a capital “A” refers to both a body of knowledge and a community of practice around software development that emerged in name at around 1999 or 2000. When you look at the early works on Agile, say, the Agile Manifesto or Kent Beck’s book, “eXtreme Programming Explained”, the focus is primarily on technical practices within software engineering (things like pair programming and test-driven development) and some organizational patterns (daily stand-ups, sitting together) that facilitate and encourage high quality software development.

For example, nearly all “strains” of Agile include starting small, and producing code in iterations, releasing frequently to customers or testers, and folding new information gained from that release into the next release. That information could be about how to improve an existing feature, or what the next highest priority feature should be. It’s all about feedback loops. Working these agile principles and tactics into your engineering practice increases the likelihood that you’ll produce software that is higher quality, more modular and thus more manageable, and as close as possible to what was requested by product managers or business analysts.

So, that’s Agile, and we’ll see in a minute how closely related it is to Lean (lean vs agile).

By “Lean Software Development”, we are primarily referring to a set of concepts and practices that originated from Mary and Tom Poppendieck in a book of that same title, which was published way back in 2003. Many other folks have certainly contributed to Lean concepts in software, but they definitely kicked us off.

If you were around the industry in 2003, you’ll recall that Agile was barely taking hold in even the most leading edge software teams. I didn’t discover the Poppendeick’s work myself until around 2008, and I’d already been practicing some version or another of Agile for several years. What’s critical about the Poppendeick’s take on Agile is that they leverage inspiration from Lean Manufacturing, and specifically the Toyota Production Process, to reframe the software development and delivery chain as akin to a value stream in a manufacturing process. It’s a very interesting and powerful contribution. Certainly there are challenges in the metaphor but many great insights as well.

This key insight, that the software your team is building is part of an overall delivery of value to an end-customer, shifts the focus of software development from one of core technical practices adhering to a spec, to one of delivering value, and thus increasing the chances of survival and growth for the enterprise as a whole (as in, if you can’t deliver value, you won’t be around long). It’s not that the Poppendieck’s devalue technical practices per se–indeed they strongly laud them–but rather they underscore the need to focus the attention of the team on delivering value to the customer.

So, if you were to imagine these two frameworks operating in tandem, you could conceive of Lean Software Development as the organizing principle for the enterprise as a whole, managing the entire value stream from “concept to cash”, as it were. And the Agile Software Development method would be the organizing principle within the engineering team’s portion of the overall value stream. They can certainly work together in this regard.

The danger of conflicted incentives arises when a focus on the delivery of high-quality software undermines (yes, it can happen) the delivery of value to the customer. A simple example would be around technical debt. Agile purists (as I myself once was, before experiencing several company failures first-hand) would push technical debt reduction to near the top of the list of priorities, which is the idea of, “clean up your code before you add anything new to it”. Clearly, though, a value-focused product manager would for push delivery of more and better features to the customer as a higher priority, even if technical debt makes that more cumbersome for the engineers. And so this is a frequent ideological battle that occurs in the planning meetings of many engineering shops on a regular basis.

There is no moral or right answer, of course. They are both right in a sense. For the product manager, doing work that doesn’t add value to the stream is essentially waste. But for the engineering team, refactoring away technical debt is a form of continuous improvement, and therefore can’t be waste. The only fair resolution is that the team have alignment around the overall mission of the company, and attempt to quantify as much as possible their specific arguments for prioritizing tech debt over new features, or the other way around, by using economic data wherever possible, and making a decision that everyone can get behind. Of course, that’s easier said than done.

Q: How is Lean Software (/Product) Development’s perspective on “Lean” different from Lean Startup?

A: In order to bring Lean Startup into the mix from lean vs agile, we need to look a little more closely at Lean Product Development historically.

For me, Lean starts with W. Edwards Deming, a statistician and quality control expert who made a name for himself by improving quality in production facilities during the war effort in WWII. He was so successful that he was sent to help Japan during its post-war reconstruction. It was there that he befriended the Union of Japanese Scientists and Engineers, with whom over a series of visits to the country he shared lots of ideas, not just around applying statistical quality controls to production, but also in developing a philosophy of product development that favored empowerment of the workers on the assembly line to make improvements to the process themselves, rather than being told how by management. This is effective because they are typically in the best position to both see causes of defects first hand, and to envision how the process could be changed to eliminate those defects. These two themes (statistical control and worker-led improvements) are a core part of the now world-renowned Toyota Production System, and by extension what we call Lean Manufacturing here in the US.

It’s worth noting that while Deming’s contributions to product development in math and statistics are profound, his more radical contributions are cultural in nature. He strongly advocated for enterprises to incentivize teams or groups over individuals, because he believed (and provided a lot of evidence to suggest) that the configuration of the overall production system has a disproportionately larger impact on the quality of production output than the contribution of any individual’s performance within that system. This notion of the favoring in the final analysis the team/system over the individual can be detected in the Agile literature as well.

The other major figure in Lean for me is Don Reinertsen, who’s probably best known for the book “Flow: Principles of Lean Product Development.” Reinertsen’s work is incredibly important, and at the same time rather hard for average reader to grasp. It’s one of those books you see increasingly referenced in slide decks at tech conferences, but you know very few of audience members have read it. It took me three times (no joke) to really grasp all of the material in it, and there are parts that still trip me up. I did have the pleasure of attending a few of Reinertsen’s workshops, so I have been helped in my understanding by hearing it straight from his mouth.

The Flow book is important because it illustrates some rather key patterns that can be found within successful Lean product development operations, and then explains the science behind why those principles actually work. Once you grasp Lean production processes on that level, it truly changes the way you see everything in an enterprise. At least it did for me.

So, for example, Reinertsen’s book stresses the critical importance of managing queues in a process, like stacks of unfinished product components between work-steps, where the vast majority of delay tends to pile up. Most tech teams aren’t used to thinking about doing knowledge work, like design or software development, in terms of a flow of value. But Reinertsen is able to show that although the product is physically invisible, it’s still subject to the same laws that exert real economic force on a factory floor. When you have a lot of half-finished inventory lying around (designs yet to be implemented, code yet to be pushed), you have holding costs accumulating, and you are accumulating cost of delay in getting feedback from the end production process for improvement.

He also expresses rather well why things like Work-In-Process (WIP) limits, a core feature of the Kanban method, are so important in managing flow. Learning Kanban, we can be told to trust that we should constrain the WIP in order to make sure that on average all tasks are completed more quickly. But it’s definitely counter-intuitive, WIP limits, and a lot of teams new to Kanban can’t quite get their heads around it. Reinertsen is able to breakdown the science of flow, queues, and WIP limits in a way that brings it home in a way that you’ll never forget. That was really big for me.

So, how does Lean Startup fit into this?

Eric Ries’ Lean Startup philosophy is essentially a mash-up of the schools of thought of Customer Development, from Steve Blank’s work, and Agile software development, sprinkled with ideas and concepts from the larger Lean literature. I was a bit critical of the Lean Startup community back in 2012 for not giving more credit to, or raising awareness of, the core Lean concepts. But in the years since then, that community has done a much better job connecting the dots for newcomers. I am very pleased to see that.

Q: Your upcoming book emphasizes teamwork aspects of Lean. Can you explain why teamwork is important?

A: Simply put, product development is a team sport. In all but the most extreme cases, products are developed by groups of people playing different roles in the production process. And in any process that depends on a number of people working together in concert, there are a few key areas that matter, such as how to manage communication both between people and out towards the wider organization, or what kind of process the team uses to coordinate their efforts, or how to make sure the goals of individuals are aligned with those of the team, and those of the team are aligned with those of the organization. In my experience working with lots of organizations, these are the main places where you get problems; communication, process, or goals misalignment. This is particularly acute in startup organizations for a couple reasons.

First, startups by nature haven’t crystallized their process yet. So there is a lot of room for failure resulting from gaps in communication. Roles are rarely as clearly defined as they are in an enterprises, and in fact you want them to be flexible at this stage. People need to wear many hats, and be able to jump around a bit in their day-to-day work. But the other side of that coin is that it can be unclear who needs to communicate with whom, how frequently, and at what level of detail. And startups that are growing, as most hope to, are adding new people to the mix all the time, making it even more difficult to manage communication protocols.

Many teams try to manage this ambiguity by simply throwing technology at it. But while using Trello, Slack, and Google Hangouts to facilitate the act of communicating certainly solves a set of important problems, it doesn’t answer the questions above about who needs to communicate with whom, when, and about what. You can’t simply tool that ambiguity away. In the book I try to illustrate some guidelines for how you can build an effective communication culture regardless of whether you’re a remote team or all in the same big room.

Second, as a process develops it tends to get more complex over time, particularly as you add people to it. As you grow, you find ways to streamline the process. Sometimes you try to do this by breaking complex tasks down into smaller, simpler ones that can be aggregated and assigned to one person for efficiency. You can see this in separating design and engineering from each other, or separating testing or quality assurance from development. It seems efficient to combine the overhead.

The problem though is that you are now introducing a larger batch size into one or another step in the process, and as Reinertsen deftly illustrates, there are severe costs that accompany increasing batch size in product development. You have to be damned sure that your costs savings in combining overhead outweigh the cost of the delays incurred from queues of half-finished work that accumulate because of larger batches. I can tell you 99% of the time it doesn’t. But most startups don’t understand the science of queues or batch sizes. If there is a single most important reason I am writing the book, it’s probably this point.

And thirdly, incentives and goals can get out of whack incredibly quickly as an organization grows in size. When your organization is two people working in the proverbial garage, it’s pretty easy to stay aligned on your goals. But as you grow, it becomes increasingly difficult to keep everyone aligned. What happens all too frequently is the development of department silos, even when you have less than a dozen people.

Imagine the scenario of a startup with three or four people, one of whom is the engineer. That engineer then hires a couple junior engineers to join the team. All of the sudden, there is an “engineering team” that is separate and distinct from the organization as a whole. They have their own language and culture of sorts, as engineers do, that is different from that of the others. You can imagine this happening over time with product and design folks as well, and then perhaps sales and marketing focused people. Before you know it, you have a 10 or 15 person company that, while they can all fit at a table at a restaurant, have already subdivided into three or four distinct cultural units within the company, whether formally recognized as such or not.

Once you have more than 5 or 6 people, and particularly if they are different seniority, the communication patterns have to be adapted to meet the needs of each person’s role and level. The junior engineer doesn’t need the same information as the senior product lead. But how do you make sure that the junior engineer has the same overall goals as the product lead, goals that are essentially transferrable from engineering to product, and that they keep in mind during their separate and very specialized day-to-day work on the product? I can tell you it is one of the hardest aspects of leading a startup. In the book I attempt to address this challenge from a variety of angles.

Q: How can people preview contents of “The Startup Patterns Book” and stay updated?

A: The best way to stay updated is by heading over to the website I set up for the book, http://startuppatternsbook.com, and signing up for the mailing list. I am writing the book in a Lean style, by writing it in small pieces and publishing those to the site to get feedback from readers. I use that feedback both to improve the sections I have already published, and to guide the overall direction of the book, in terms of which topics people are more or less excited about.

As of this writing, I have completed about half of the first of what will likely be four chapters. I am aiming to have some printed version of the book available for purchase or download or something by the end of 2015.

You can sign up to the Startup Patterns mailing list and receive the latest updates here.