The untested surgical team?

In reading Fred Brooks' seminal The Mythical Man Month, I found myself intrigued by his description of the "surgical team" programming group in chapter 3. Especially when he argues:

"Managers need to be sent to technical refresher courses, senior technical people to management training...Whenever talents permit, senior people must be kept technically and emotionally ready to manage groups or delight in building programs with their own hands...The whole notion of organizing surgical-type programming teams is a radical attack on this problem. It has the effect of making a senior man feel that he does not demean himself when he builds programs, and it attempts to remove the social obstacles that deprive him of that creative joy. Furthermore, that structure is designed to minimize the number of interfaces...It is really the long-run answer to the problem of flexible organization." (120)

This post does a pretty good job summarizing, though I think its conclusions are too strong. My response is:

I am reading The Mythical Man Month now, and was curious as to whether this approach has been successfully studied in the academic literature. Aside from Brooks' book, no Google Scholar papers seem to try it, which I find interesting.

I agree with you that, since 1972, many of these roles have been automated...but I think you may be quickly dismissing a few of them.

In every modern programming language (Python, Groovy, Ruby, C#, and yes, even Java and C++) there are idiomatic ways of writing certain snippets of code, and ways that work but are "sub-optimal."

For example, many modern techniques using LINQ in .NET can turn 10 line filter a list loops into one-liners. Under the assumption that fewer readable lines of code for the same functionality = greater maintainability, these optimizations would be well to strive for. Hence, a "language lawyer", i.e. That Guy That Knows the Language Inside-Out, may be useful.

I think, when it comes to tools, you're thinking of IDEs, etc. We have seen a dramatic rise of those. But most programming teams still have specialized libraries that are developed internally and used from project to project. For example, that one Active Directory security library that is written once and reused from project to project. I think this is more of what the Toolsmith is supposed to do, in terms of
"building specialized programming tools."

For example, in many modern projects I have the arduous task of migrating data from one system to another. This often involves developing a mediator schema or some type of transform. I know the right thing to do is to build a script, just in case I have to repeat the process or add some more data later. But if I'm focusing on Building Shiny New System X, the time I take to write a program to go from Old Piece of Crap Y to X is time taken away from polishing X. That's what a toolsmith would be for.

As for administrators, editors, and secretaries, I'm reminded of Joel Spolsky's work on the Development Abstraction Layer. I think such roles are important, for exactly the reasons he described.

Chief Programmer/co-pilot sounds very much like modern XP Pair Programming, with a very clear Sr/Jr relationship. While TDD/BDD may have them writing their own tests, one can argue that a Tester would still be beneficial for functional testing, and writing al that "glue code" to make testing easier (mock objects, dummy records, etc etc).

I guess my point is, while automation is definitely compelling, don't be so quick to dismiss his ideas. I find myself very surprised that this organizational strategy hasn't been put more to the test.

I would agree that program clerk is a relic of the date of publication, easily replaced by source control.

I find it curious that Brooks' assertions don't seem to have been put to the test in the literature since 1972. I wonder if any case studies have been done regarding this approach?

Also, regarding my assertion that "the assumption that fewer readable lines of code for the same functionality = greater maintainability", I remember studying metrics in Software Project, Process, and Quality Management at Arizona State University...but I don't ever remember running into a heuristic such as this.

Intuitively, it makes sense. A caveat must be made for Perl, of course, which can have one obnoxious line that does everything but is impossible to understand, hence why I included "readable" in the assertion. I wonder if something similar exists in the literature?

If not, it'd be fun to make this Vaidyanathan's Law. (Not to mention deliciously ironic, as I'm all about prolix tautology whereas the law argues for brevity! >:D)

Architecture as defined in Brooks' Aristocracy, Democracy, and Systems Design (The Mythical Man-Month)

"By the architecture of a system, I mean the complete and detailed specification of the user interface...The architect of the system, like the architect of a building, is the user's agent. It is his job to bring professional and technical knowledge to bear in the unalloyed interest of the user, as opposed to the interests of the salesman, the fabricator, etc." (45)

I don't know that I've ever seen a system's architecture defined that way before. It seems limiting to think of architecture solely as the user interface, but the idea of the architect as the user's agent is interesting.

"Architecture must be carefully distinguished from implementation. As Blaauw has said, 'Where architecture tells what happens, implementation tells how it is made to happen.' He gives as a simple example a clock, whose architecture consists of the face, the hands, and the winding knob. When a child has learned this architecture, he can tell time as easily from a wristwatch as from a church tower. The implementation, however, and its realization, describe what goes on inside the case--powering by and any of many mechanisms and accuracy control by any of many." (45)

A great example of the difference between architecture and implementation.

Programming teams as surgical teams

An interesting idea from Fred Brooks' The Mythical Man-Month:

"At computer society meetings one continually hears young programming managers assert that they favor a small, sharp team of first-class people, rather than a project with hundredsof programmers, and those by implication mediocre. So do we all. But this naive statement of the alternatives avoids the hard problem--how does one build large systems on a meaningful schedule?" (30)

"In one of their studies, Sackman, Erikson, and Grant were measuring performance of a group of experienced programmers. Within just this groug the ratios between best and worst performances averaged about 10:1 on productivity mesurements and an amazing 5:1 on program speed and space measurements! In short, the $20,000/year programmer may well be 10 times as productive as the $10,000/year one. The converse may be true, too. The data showed no correlation whatsoever between experience and performance. (I doubt if this is universally true.)" (30)

"I have earlier argued that the sheer number of minds to be coordinated affects the cost of the effort, for a major part of the cost is communication and correcting the ill effects of miscommunication (system debugging). This, too, suggests that one wants the system to be built by as few minds as possible...the problem with the small, sharp team concept: it is too slow for really big systems." (30-31)

"The dilemma is a cruel one. For efficiency and conceptual integrity, one prefers a few good minds doing design and construction. Yet for large systems one wants a way to bring considerable manpower to bear, so that the product can make a timely appearance. How can those two needs be reconciled?" (31)

"Mills proposes that each segment of a large job be tackled by a team, but that team be organized like a surgical team rather than a hog-butchering team. That is, instead of each member cutting away on the problem, one does the cutting and the others give him ever support that will enhance his effectiveness and productivity." (32)

"The surgeon. Mills calls him a chief programmer. He personally defines the functional and performance specifications, designs the program, codes it, tests it, and writes its documentation. He writes in a structured programming language such as PL/I, and has effective access to a computing system which not only runs his tests but also stores the various versions of his programs, allows easy file updating, and provides text editing for his documentation. He needs great talent, ten years experience, and considerable systems and application knowledge, whether in applied mathematics, business data handling, or whatever.

The copilot. He is the alter ego of the surgeon, able to do any part of the job, but is less experienced. His main function is to share in the design as a thinker, discussant, and evaluator. The surgeon tries ideas on him, but is not bound by his advice. The copilot often represents his team in discussions of function and interface with other teams. He knows all of the code intimately. He researches alternative design strategies. He obviously serves as insurance against disaster to the surgeon. He may even write code, but he is not responsible for any part of the code.

The administrator. The surgeon is boss, and he must have the last word on personnel, raises, space, and so on, but he must spend almost none of his time on these matters. Thus, he needs a professional administrator who handles money, people, space, and machines, and who interfaces with the administrative machinery of the rest of the organization. Baker suggests that the administrator has a full-time job only if the project has substantial legal, contractual, reporting, or financial requirements because of the user-producer relationship. Otherwise, one administrator can serve two teams.

The editor. The surgeon is responsible for generating the documentation--for maximum clarity he must write it. The is true of both external and internal descriptions. The editor, however, takes the draft or dictated manuscript produced by the surgeon and criticizes it, reworks it, provides it with references and bibliography, nurses it through several versions, and oversees the mechanism of production.

Two secretaries. The administrator and the editor will each need a secretary; the administrator's secretary will handle project correspondence and non-product files.

The program clerk. He is responsible for maintaining all the technical records of the team in a programming-product library. The clerk is trained as a secretary and has responsibility for both machine-readable and human-readable files. All computer input goes to the clerk, who logs and keys it if required. The output listings go back to him to be filed and indexed. The most recent runs of any model are kept in a status notebook; all previous ones are filed in a chronological archive. Absolutely vital to Mills' concept is the transformation of programming 'from private art to public practice' by making all the computer runs visible to all team members and identifying all programs and data as team property, not private property. The specialized function of the program clerk relieves programmers of clerical chores, systematizes and ensures proper performance of those oft-neglected chores, and enhances the team's most valuable asset--its work product. Clearly the concept as set forth above assumes batch runs. When interactive terminals are used, particularly those with no hard-copy output, the program clerks functions do not diminish, but they change. Now he logs all updates of team progress copies from private working copies, still handles all batch runs, and uses his own interactive facility to control the integrity and availability of the growing product.

The toolsmith. File-editing, text-editing, and interactive debugging services are now readily available, so that a team will rarely need its own machine and machine operating crew. But these services must be available with unquestionably satisfactory response and reliability; and the surgeon must be the sole judge of the adequacy of service available to him. He needs a toolsmith, responsible for ensuring this adequacy of basic service and for constructing, maintaining, and upgrading special tools--mostly interactive computer services--needed by his team. Each team will need its own toolsmith, regardless of the excellence and reliability of any centrally provided service, for his job is to see to the tools needed or wanted by his surgeon, without regard to any other team's needs. the tool-builder will often construct specialized utilities, cataloged procedures, macro libraries.

The tester. the surgeon will need a bank of suitable test cases for testing pieces of his work as he writes it, and then for testing the whole thing. The tester is therefore both an adversary who devises system test cases from the functional specs, and an assistant who devises test data for the day-by-day debugging. He would also plan testing sequences and set up scaffolding required for component tests.

The language lawyer. By the time Algol came along, people began to recognize that most computer installations have one or two people who delight in mastery of the intricacies of a programming language. And these experts turn out to be very useful and widely consulted. The talent here is rather different from that of the surgeon, who is primarily a system designer and who thinks representations. the language lawyer can find a neat and efficient way to use the language to do difficult, obscure, or tricky things. Often he will need to do small studies (two or three days) on good technique. One language lawyer can service two or three surgeons.

This, then, is how 10 people might contribute in a well-differentiated and specialized roles on a programming team built on the surgical model."

" (34)

"Notice in particular the differences between a team of two programmers conventionally organized and the surgeon-copilot team. First, in a conventional team the partners divide the work, and each is responsible for design and implementation of part of the work. In the surgical team, the surgeon and copilot are each cognizant of all of the design and all of the code...Second, in the conventional team the partners are equal, and the inevitable differences of judgment must be talked out or compromised. Since the work and resources are divided, the differences in judgment are confined to overall strategy and interfacing, but they are compounded by differences of interest...In the surgical team, there are no differences of interest, and differences of judgment are settled by the surgeon unilaterally."(34)

Obviously some of these concepts are dated and can be replaced by computer functions, such as the program clerk. But has this style of teach ever been effectively implemented in the literature?

Bottom Up and Top Down Software Development

Software Engineering is a rapidly changing beast, as software itself, being made of "pure thought stuff" (Brooks, Mythical Man Month) rapidly changes. Not only has the process orientation of the industry changed, from cascading waterfalls to Agile methodologies (Agile Manifesto, Beck et al), but the design focus has changed.

In 1980s, design and architecture tended towards the development of large models that fully specified the software. The emphasis on modeling and architecture led to favoring code generation and a focus more on the high level "how things work." Tools like IBM Rational Rose were seen as holistic tools that could allow full specification of the software process from the ground up.

These tools have gained limited market share. As part of the Agile Manifesto's emphasis on "Working software over comprehensive documentation," the emphasis on system specification via UML modeling and up-front specification has given way to TDD development (Beck, Test Driven Development by Example). This code-first approach has found wide acclaim with software developers and industry, and even design is being driven by it's ramifications via the use of Mock Objects as a design tool. (Freeman et all, "Mock Roles Not objects" http://static.mockobjects.com/files/mockrolesnotobjects.pdf)

The basic TDD workflow is fairly simple: write a test that fails, write just enough code to make it pass, refactor and evolve. This approach has won acclaim because it breaks the intractible software system into manageable chunks. But why hasn't modeling received similar treatment?

In theory, it's just as easy to write a test and make it pass as to write a class diagram for a single class, then implement it. Write a failing test for a new method, add it to the class diagram, then add sequence diagram that stubs that class and that method.

Why isn't this done more? Is it poor support from the tools, i.e. it's hard to quickly create a modeling artifact and bind it to the project in the same way it is for tests? Is it lack of developer buy-in, i.e. models are completely unnecessary? Is it a perception of complexity of the modeling constructs themselves, i.e. the models are useful, but they're too complex and add little value?