Robust Web Application Charting in Grails

So I'm working on a web-app development project where my customer wants some nice "dashboard" charts for the status of pieces in the system. I'm building it in Grails (which is t3h s3x.), and my initial approach was to use the JFreeChart library. I've used JFreeChart before in school J2SE development projects, and was familiar with its API, so it seemed like it would be a good alternative.

Using JFreeChart is really easy in Grails. Here's a quick recipe:

1. Download JFreeChart and JCommon jar files
   
2. Place the jar files in the "lib" directory of your grails-app
3. When you want graphs, import them and use them as regular jar files, like so:

Using JFreeChart in a Utility class

I find that it's generally good practice to encapsulate dependencies, so I created a class to wrap drawing that essentially follows the example given in the JFreeChart source for creating a dial chart. Then, I created a service to handle charting, since that is something that I expect to use in many controllers (DRY is good)

Creating a Service to manage drawing

Using this service in my controllers becomes really easy.

Controller

View

This was all well and good...until I viewed the images. They're ugly. Pug fugly. Like, ugly ugly.

So what's a boy to do? Well, I found a cute little library called HighCharts. Its graphs are sexy. They're fairly easy to make, too. So I built one. It was cool. Now I was going through the odious task of removing JFreeChart. Why?

1. its added a lot of bloat to the code
   
2. it's not good to mix approaches for doing the same thing in different places

But it's an "odious" task and I'm loathe to do it, because the charts generated "work." They're just "not pretty", which can probably be adjusted with some parameters and a deeper dive into the JFreeChart Theming API. It's just extra work, and this solution seems to work better...

Just as I was about to remove JFreeChart, I realized something: I had switched my charting approach from server-side to client-side. Good for processing. The client-side implementation relies on Javascript, of course. What if it was turned off? That could be a 3am support call...so should I do everything in JFreeChart?

Until I remembered my old friend, the <noscript> tag. If I use that, I have a configurable approach. If Javascript is enabled, make a HighCharts chart. If not, I still have my old server side chart ready to go.

One could argue that worrying about Javascript being turned off in this day and age is silly. Plus, managing both libraries and mechanisms for chart drawing is a maintainability concern moving forward. But in so doing I get some functional robustness.

I'm fairly certain the trade-off between robustness and efficiency/ease of maintainability is fairly common. The correct resolution of the tension of these forces depends on the requirements. Nevertheless, it's a good thing to look out for:

If your server side code can avoid doing stuff, avoid it by doing it client side. But keep the flexibility in there in case the client side can't.

CSE360 syllabus rev 0.2

Introduction to software engineering topics outline:
8 weeks

Schedule:
week 1: History (Read : The Mythical Man Month. 5 page reports due class 1 next week )
-- 1960s : We Have a Problem...60% of software fails.
-- 1970s : Focus on Process! (origins and evolution of project management.)
-- 1980s : Focus on tools! (Rational Rose, code generation)
-- 1990s : Focus on People! (Agile Manifesto. Evolution of WWW. .Com bubble)
-- 2000s : Focus on evolution! (frameworks, design patterns, continuous integration, source control becomes standard, the software craftsman movement)
-- 2010s : The Brave New World (Mobile, Mesh, Cloud, etc)

Case studies/scenarios
Therac-25
Nasa Rover
Google
Windows Vista

week 2: Process models, Testing (TDD, Unit, Functional) introduce project
(Read : The Pragmatic Programmer. 5 page reports due class 1 next week )

Introduce Agile methodology.
Discuss project.
Introduce a "Planning Game"
Introduce NUTs (Nebulous Units of Time -- brief discussion on estimation techniques)
Introduce Source control, CI, bug tracking, and deployment process
Form teams
Extra topics:
"a discussion and discourse of how to build software from start to finish. Various topic matter falls under this umbrella which includes"
* CMMI
* ISO/IEEE process standards
* Software Metrics
* Analyzing/Managing/Mitigating Risk"


week 3: Source Control tools (versioning, continuous integration) form teams
Team led tutorial setup of :
Mercurial
SVN (can use Google Project Hosting or local box for tutorial. Project Hosting required for course)
Git (GitHub)
Tutorial about the source control methodologies themselves:

In class Flame War/Debate
This is an activity where the groups that researched the source control tools will lead an argument as to why it's obviously better and more correct than yours
Iteration 1:
Implement "Hello world"
Add authorization and authentication

week 4: Requirements analysis (BDD, TDD, Mock Objects, CRC cards, Joel on Software Functional Specs)
Discussion of TDD/BDD/Domain Driven Design
Discussion of SRS/Functional Spec/CRC cards
Flame War/Debate
This is an activity where the groups that researched the functional requirements methodologies will lead an argument as to why it's obviously better and more correct than yours

Iteration 2:
Implement functional spec

Write behavioral tests using mock objects for domain model that stores teachers, students, and profiles (easyb)

Write unit tests for CRUD teachers, students, profiles. Users can modify own profile but not other profiles. Admin can modify all.

week 5: Design (UML, design patterns, MVC)
Discussion of basic UML (class diagram, activity diagram, sequence diagram)
Discussion of Design Patterns
-- Group Led Presentations on
Strategy
Factory
Template Method
Observer
Command
-- I will discuss
Decorator
Mediator
Facade
Software Architecture -- Pipe & filter, Blackboard, MVC

Iteration 3: revise domain model.
Add lessons and scheduling.
Refactor standard grails views to use AJAX.
Develop different compensation Strategies (pay in full before session.Pay half up front half after. Pay all after)
Develop a Lesson Factory
Use Command Objects to encapsulate validation and pass parameters to controllers

week 6 & 7: Implementation (In class pairing sessions. We may get a lab. Otherwise, bring your laptops!)
Extreme Programming/Scrum/Lean
Self documenting code
red light/green light/refactor/repeat
code analysis (CodeNarc)
Documentation (Doxygen)

Iteration 4:
Integrate 3rd party plugins to add
Audit Logging
Full-text Search for lesson material, profile content
Graphs and charts for teachers to see number of students / week, number of lessons / month, revenue made, time spent
Investigate use of external solutions to integrate real-time chat functionality
Red5, Adobe Air, OpenLazlo, others?

week 8: Testing of other team's projects / wrap up
Security penetration Testing
Load Testing
other misc topics...


Textbooks / Resources:
-- Fred Brooks The Mythical Man Month (Required)
-- Dave Hunt and Andy Thomas The Pragmatic Programmer (Required)
-- Grails in Action (Optional)
-- Head First Object Oriented Analysis and Design & Head First Design Patterns (Optional)
-- Joel on Software
-- Coding Horror
-- Software Engineering Radio
-- Grady Booch on Architecture

Books you should read to be awesomesauce instead of Lame McSlame:
-- Patterns of Enterprise Application Development by Martin Fowler
-- Refactoring by Martin Fowler, Kent Beck, et al
-- Design Patterns by GoF
-- Test Driven Development by Kent Beck
-- Clean Code by Robert Martin
-- Domain Driven Design by Eric Evans

Contrast to traditional CSE360 courses:
Waterfall - I'm not even going to teach it to you, because it's wrong.
Your only responsibility is to know that it exists.
When someone brings it up, you should spit in disgust.
Some people argue that on small projects it's ok.
They're wrong. I've yet to see a meaningful project that doesn't require iterative refinement.

Spiral -- A historical method. We'll talk about it a bit, since Rational Unified Process and every proceeding Agile method is basically Spiral on steroids.

Project:
We're going to work from an Agile methodology, using Test Driven Development, to build a social networking web application where users can connect in real-time.
Many classes like this allow you to choose your own implementation technology, and do the entire project from soup to nuts.

In my own experience, we built the backend of a restaurant management system.
We did no coding until the month before it was due, then we choose to use Visual Studio because it made GUI creation easy. We spent 2 weeks learning the language and basically had 2 all-nighters where we cranked out the components. I will not allow you to repeat that process.

Furthermore, it doesn't reflect what happens in the real world. Most of you will get a job at a company where you will already have prescribed implementation technologies. You will probably be supporting an existing code base (in which case you should definitely read Refactoring), which will need maintenance while you add new features. Consequently, I'm going to constrain you a little bit. This actually makes your life easier.

You will use Grails to build a social networking application where users can connect and have real-time video conferencing. Think Facebook meets ChatRoulette.


-- Resources
Peter Ledbrook & Glen A. Smith's Grails in Action
Scott Davis Mastering Grails IBM Developer Works series
Grails.org documentation
W3schools.org
Head First HTML/XHTML/CSS

-- Expectations
If you have not developed web applications before, you'll have the joys of learning HTML/JS/CSS (the Assembly of the web) real quick.

All of you know Java from CSE110/CSE205.
Groovy is a "psuedo" superset of Java that incorporates lessons learned from dynamic languages like Ruby and Python.
Grails is an incredibly powerful web application framework built in Groovy.
You will learn these technologies. Quickly.

You will use a source control system. We will have mini tutorials for setup of SVN, git, and Mercurial. You will lead them.

You will achieve at least 90% code coverage on your tests.

You will use code analysis tools to diagnose implementation issues.

You will use a continuous integration server environment. Jenkins http://jenkins-ci.org/

You will deploy new stable versions of the application "to production" every week.

Your code will be reviewed. I will savagely hunt down lame code form the depths of obscurity and shine the light of critique on it. Every time you write dirty.hack I expect a multi-line comment apologizing for it and rationalizing why you had to do it that way. The codebase itself better not be a patchwork set of hacks. We're practicing the Pragmatic CARE About Your Craft in this course.

My goal is for you to be able to walk into a job interview and impress the hell out of your interviewers.

I want you to know all of the latest and greatest technologies, likely know at least some of the tools they're already using, and have some insights and ideas about how they could do things better based on the current state of best practices as identified by your studies.


I cannot teach you everything that you need to know to be a good software engineer.
But I vow that you will not have the experience that many of my friends have experienced, where you go into industry and know none of the tools and have only academic knowledge about how to build enterprise, _professional quality_ software.

Disclaimer:
If you want to snooze through an easy A, this is not the course for you...That said, there will be _plentiful_ extra credit opportunities.

I *want* to give you all As. But you're going to have to _earn_ it.

If you expect a boring and dry lecturer, look elsewhere.

If you expect to sit around and never answer questions, this is not the course for you.

You will all be expected to bring name tents, every session. No name tent = loss of points. I will call on you.

Also, in the words of dhh (fist bump if you know who I mean), I'm a R-rated Individual. You're all junior level students in a CSE curriculum, i.e. adults. If you're easily offended, you may not like me. Your call.

I've gone back and forth about laptops in class. Final decision: No Internet. I know that when I'm not engaged, the first thing I do is go to Facebook/check my e-mail. It's natural. But I'm an asshole. This is a wi-fi jammer. When we're in class this is on.

I will also be walking around the room. If you're on Minesweeper or watching something else, I'll say "thanks for coming. We'll see you next week."

No phones, obviously.

Grading
- The class grade comes out of 1000 points.
- You will be graded based on your contributions to the project. Each of you will be expected to contribute in every step of the way.
For the project, you will have to prepare:
-- A functional spec (no dated SRS documents here. We're going Agile with something similar to the Spolsky document) 200 pts
-- A set of UML design diagrams (Class, Use Case, possibily Activty) 200 pts
-- On Sunday night every week I will be checking out your code from source control and reviewing your progress.

400 pts / 8 wks = 50 pts / week.
10 pts/ feature spec.
10 pts for test.
10 pts for implementation.
10 pts for analysis.
10 points for refactoring (optimization for performance, clarity, elegance. Adequate documentation on code produced)

You will be graded based on your NON-TRIVIAL commits. All of you will be expected to write tests, design, feature code, and documentation.
-- You will write a few short essays. You will write reflections on The Mythical Man Month, The Pragmatic Programmer,
and you will produce critiques on one another's designs and other work artifacts. #protip: "It's cool" won't cut it.

200 pts = 2 x 100 pts essays (The Mythical Man Month and The Pragmatic Programmer)
100 pts / 8 wks = 20 pts / week (collaborative code review/critique/analysis)
(>_>)..."wait a minute", you say, "20 pts / week != 100 pts!"
Correct. This is why I'm a Software Engineer, not a Theoretical Computer Scientist. I can't do math.

...Actually, it's because critique and analysis are higher order cognitive functions.
Novices/Advanced Beginners will struggle with this, but this is how we ascend Bloom's Taxonomy and acheieve expertise.
You are not experts. But you can try hard. Hopefully, in those attempts you will create new neural pathways and channel into some Higher Order Ideas.
But you're not expected to blow me away, though I always respond well when I am blown...away.
...(In other words: PSSST, THERE'S EXTRA CREDIT HERE)


Read the whole Mythical Man Month in a week? Then the whole Pragmatic Programmer the week after? Yeah buddy.
Honors students in the liberal arts get asked to do that all the time.
If you're going to be a professional Computer Scientist you should be at least as smart as some pretentious kid studying ancient Greek literature that sips lattes and reads the New York Times daily. ;P
(Relax, I'm just teasing! We all love hipsters/'anti-hipster intelligentsia' with their little chai tea lattes, Apple products, and panache for pretty services like Facebook and tumblr. )

Class Philosophy:
"You don't snuggle with Max Power! You STRAP YOURSELF IN AND FEEL THE Gs!" -- Homer Simpson
...buckle up, baby. This is gonna be fun. >:D

Tentative course plan for first course taught

Introduction to software engineering topics outline:
8 weeks

week 1: History
week 2: Process models, Testing (TDD, Unit, Functional) introduce project
week 3: Source Control tools (versioning, continuous integration) form teams
week 4: Requirements analysis (BDD, CRC cards, Joel on Software Functional Specs)
week 5: Design (UML, design patterns, MVC)
week 6: Implementation? (In class XP session. We may get a lab. Otherwise, bring your laptops!)
week 7: Testing of other team's projects
week 8: wrap up

Textbooks / Resources:
-- Fred Brooks The Mythical Man Month (Required)
-- Dave Hunt and Andy Thomas The Pragmatic Programmer (Required)
-- Grails in Action (Required)
-- Head First Object Oriented Analysis and Design & Head First Design Patterns (Optional)
-- Joel on Software
-- Coding Horror
-- Software Engineering Radio
-- Grady Booch on Architecture

Books you should read:
-- Patterns of Enterprise Application Development
-- Refactoring
-- Design Patterns
-- Test Driven Development
-- Clean Code
-- Domain Driven Design

Waterfall - I'm not even going to teach it to you, because it's wrong.
Your only responsibility is to know that it exists.
When someone brings it up, you should spit in disgust.
Some people argue that on small projects it's ok.
They're wrong. I've yet to see a meaningful project that doesn't require iterative refinement.

Spiral -- A historical method. We'll talk about it a bit, since Rational Unified Process and every proceeding Agile method is basically Spiral on steroids.

Project:
We're going to work from an Agile methodology, using Test Driven Development, to build
a social networking web application where users can connect in real-time.
Many classes like this allow you to choose your own implementation technology, and do the entire project from soup to nuts.
In my own experience, we built the backend of a restaurant management system.
We did no coding until the month before it was due, then we choose to use Visual Studio because it made GUI creation easy.
We spent 2 weeks learning the language and basically had 2 all-nighters where we cranked out the components.
I will not allow you to repeat that process.

Furthermore, it doesn't reflect what happens in the real world. Most of you will get a job at a company where you will already have prescribed
implementation technologies. You will probably be supporting an existing code base (in which case you should definitely read Refactoring), which will
need maintenance while you add new features. Consequently, I'm going to constrain you a little bit. This actually makes your life easier.

You will use Grails to build a social networking application where users can connect and have real-time video conferencing. Think Facebook meets LiveJasmin.
(Don't pretend you don't know what I'm talking about ;P)

-- Resources
Peter Ledbrook & Glen A. Smith's Grails in Action
Scott Davis Mastering Grails IBM Developer Works series
Grails.org documentation
W3schools.org
Head First HTML/XHTML/CSS

-- Expectations
If you have not developed web applications before, you'll have the joys of learning HTML/JS/CSS (the Assembly of the web) real quick.
All of you know Java from CSE110/CSE205.
Groovy is a "psuedo" superset of Java that incorporates lessons learned from dynamic languages like Ruby and Python.
Grails is an incredibly powerful web application framework built in Groovy.
You will learn these technologies. Quickly.
You will use a source control system. We will have mini tutorials for setup of SVN, git, and Mercurial. You will lead them.
You will achieve at least 90% code coverage on your tests.
You will use a continuous integration server environment.
You will deploy new stable versions of the application "to production" every week.
YOUR CODE WILL BE REVIEWED. If it is shit, I _will_ call you out on it. In class. You'll get to explain why saw fit to write Crap McScript to achieve unmaintainable functionality that belongs on ___
When you are done with my class, you will walk into a job interview and they will ask you "Are you a good software engineer?", you will say "I'm not just a good software engineer, I AMZ t3h SEX!" and proceed to wow the pants off of them.

I cannot teach you everything that you need to know to be a good software engineer.
But I vow that you will not have the experience that many of my friends have experienced, where you go into industry and know none of the tools and have only academic knowledge about how to build enterprise, _professional quality_ software.
If you want to snooze through an easy A, this is not the course for you.
That said, there will be _plentiful_ extra credit opportunities.
I *want* to give you all As. But you're going to have to _earn_ it.
If you expect a boring and dry lecturer, look elsewhere. If you expect to sit around and never answer questions, this is not the course for you.
You will all be expected to bring name tents, every session. No name tent = loss of points. I will call on you.
Also, in the words of dhh (fist bump if you know who I mean), I'm a R-rated Individual. You're all junior level students in a CSE curriculum, i.e. mature adults.
If you're easily offended, you may not like me. Your call.
I've gone back and forth about laptops in class. Final decision: No Internet. I know that when I'm not engaged, the first thing
I do is go to Facebook/check my e-mail. It's natural. But I'm an asshole. This is a wi-fi jammer. When we're in class this is on.
I will also be walking around the room. If you're on Minesweeper or watching something else, I'll say "thanks for coming. We'll see you next week."
No phones, obviously.

Grading
- You will be graded based on your contributions to the project. Each of you will be expected to contribute in every step of the way.
For the project, you will have to prepare:
-- A functional spec (no dated SRS documents here. We're going Agile with something similar to the Spolsky document)
-- A set of UML design diagrams (Class, Use Case, possibily Activty)
-- On Sunday night every week I will be checking out your code from source control and reviewing your progress.
You will be graded based on your NON-TRIVIAL commits. All of you will be expected to write both tests and feature code.
-- You will write a few short essays. You will write reflections on The Mythical Man Month, The Pragmatic Programmer,
and you will produce critiques on one another's designs and other work artifacts. #protip: "It's cool" won't cut it.


Read the whole Mythical Man Month in a week? Yeah buddy. Honors students in the liberal arts get asked to do that all the time. If you're going to be a professional
Computer Scientist you should be at least as smart as some pretentious kid studying ancient Greek literature that sips lattes and reads the New York Times daily.

"You don't snuggle with Max Power! You STRAP YOURSELF IN AND FEEL THE Gs!" -- Homer Simpson
...buckle up, baby. >:D

Why Political Informatics is necessary, and the value of open standards

On 2/7/2011, Maricopa county will have a tax lien online action at http://www.bidmaricopa.com. This auction will serve to generate money for the county by giving private investors a chance to buy relatively low-risk investments via a public auction. Arizona has passed laws to make all of this information, including the properties info, available via "public notices."

So they've gone ahead and published the data in a newspaper, and also "posted it on the web."

I use quotations because their definition of "posting on the web" is very loose:

http://www.publicnoticeads.com/AZFRAME/search/view.asp?T=PN&id=14\1182011_15476677.HTM
http://www.publicnoticeads.com/AZFRAME/search/view.asp?T=PN&id=14\1182011_15476677.HTM

Utterly repugnant. The "disclaimer" at the top suggests: "NOTE: Some notices are extracted from PDF files and may be difficult to read."

...which is some jerky's way of playing Cover Your Ass because they couldn't be bothered to list the information in a way that might make it usable to the public. This follows the letter of the law, but certainly not the intent. In order to use this "posting", it is up to individuals to read through a wall of text and try to figure out how many records are in there, what all the information corresponds to, and how to use that information to make an informed decision.

"Oh, you want some basic questions answered on the different types of items available? Here's a 500 page dump of garbage that has it all in there somewhere. In theory. Have fun."

Appalling.

Luckily, as a Computer Scientist I'm able to read the header and recognize something:

"THE INFORMATION FOR EACH PARCEL ADVERTISED HEREIN IS PRINTED IN THE FOLLOWING ORDER: ONE UP SEQUENCE NUMBER, WHICH WILL BE USED AS AN IDENTIFIER FOR THE PURPOSE OF ADMINISTERING THE SALE; TAX AREA CODE; PARCEL NUMBER; OWNER; SEC., TWN., AND RNG. OR LOT, BLK AND TRACT (BETWEEN ASTERISKS); DESCRIPTION: VALUE OF REAL ESTATE; VALUE OF IMPROVEMENTS; VALUE OF PERSONAL PROPERTY; AND AMOUNT OF TAXES DUE BEFORE INTEREST AND FEES."

This is a structured document. Had this information been stored in an open and well-established semantic format like the Extensible Markup Language (XML), it would be trivial to automatically extract the information from it (we call this "parsing"), using a standard tool. In fact, we could even transform the data (using XSLT) into a web page that displayed it in an easy to read and usable format.

Instead, some yokel did a Select All->Copy/Paste on a nicely formatted PDF file (which apparently they don't want to release to the public) and dumped some textual diarrhea on a page for some poor saps to sift through. One can imagine the common person's approach would be to go line by line and try to manually remove the data, which would be effort and time intensive.

It's more than just presentation. If this data is stored in a nicely structured format, we can query it to ask intelligent questions. What if I wanted to find all the plots with liens under $2000? All the plots with liens that have been paid before? All the plots owned by investors/companies versus private individuals? I can't do any of that with the data in this format.

All is not lost. As a computer scientist and informatician, I know that there are tools for information extraction from text that can help me. The most powerful one is called a Regular Expression, which I can use to perform targeted pattern matching on a block of text. If I figure out the right regular expression, I can parse the text and output it in a more readable format.

Figuring out such an expression is non-trivial. My annoyance is increased by realizing that this step will essentially duplicate work: I must take data that was nicely structured somewhere, then callously dumped in a big pile of blah, and reconstruct the structure. This duplication of effort wastes time and energy.

At least Yavapai County provides the PDF: http://www.publicnoticeads.com/notices/AZ/22/pdfs/YC%20TaxSaleAdvList2011final.pdf

...and that still sucks. Because while that may be easier to read for the eyes, PDF is much more complex to work with for automated information extraction than other formats. So querying still requires human brain q/a and is an inefficient process.

If they really cared about making this information "publicly accessible", they would store it in a structured and easily readable format on their websites for common users. They would also provide web service endpoints that could be used by external programs to gather the data and query it.

"Oh, that sounds expensive. Hard. Time consuming. Given AZ's budget problems-- where they're so strapped for cash they choose to let transplant patients die while they fund governmental initiatives to build bridges for endangered squirrels-- they simply don't have the budget to do that."

No. It's not. Furthermore, whatever small amount of resources would be necessary to accomplish such a task would be more than made up for by cost-savings in time and effort to prepare these documents, fax them to newspapers, and make countless duplicates of information that should live in one place. Investment that actually improves efficiency and costs less in the long run.

Besides, last I checked there's this small school called Arizona State University (and the University of Arizona, plus many other community colleges) that has Computer Science and Computer Information Systems students. If the costs were so prohibitive, perhaps they could approach the university as resources and try to have students give them FREE WORK.

Instead, they're busy cutting already low educational expenditure levels to even lower rates. Taking money away from the places that could help us all. Is that a sign that "all government is inefficient", or just really poor leadership?

While we ponder the answer to that question, I'm going to write a web app.

Video Games offer the most effective form of story telling and learning

Thesis: Video games offer the most effective form of story-telling in the modern experience. This is because video games offer interactivity, sensory immersion across the 3 most information carrying senses (vision, sound, touch), and the duration of time spent playing the common video game is longer than the amount of time spent with other media (movies, small novels).

Examples of video games that can teach
- Dance dance revolution, XBox Kinect game
- Brain Age

...elaborate more later

The untested surgical team?

In reading Fred Brooks' seminal The Mythical Man Month, I found myself intrigued by his description of the "surgical team" programming group in chapter 3. Especially when he argues:

"Managers need to be sent to technical refresher courses, senior technical people to management training...Whenever talents permit, senior people must be kept technically and emotionally ready to manage groups or delight in building programs with their own hands...The whole notion of organizing surgical-type programming teams is a radical attack on this problem. It has the effect of making a senior man feel that he does not demean himself when he builds programs, and it attempts to remove the social obstacles that deprive him of that creative joy. Furthermore, that structure is designed to minimize the number of interfaces...It is really the long-run answer to the problem of flexible organization." (120)

This post does a pretty good job summarizing, though I think its conclusions are too strong. My response is:

I am reading The Mythical Man Month now, and was curious as to whether this approach has been successfully studied in the academic literature. Aside from Brooks' book, no Google Scholar papers seem to try it, which I find interesting.

I agree with you that, since 1972, many of these roles have been automated...but I think you may be quickly dismissing a few of them.

In every modern programming language (Python, Groovy, Ruby, C#, and yes, even Java and C++) there are idiomatic ways of writing certain snippets of code, and ways that work but are "sub-optimal."

For example, many modern techniques using LINQ in .NET can turn 10 line filter a list loops into one-liners. Under the assumption that fewer readable lines of code for the same functionality = greater maintainability, these optimizations would be well to strive for. Hence, a "language lawyer", i.e. That Guy That Knows the Language Inside-Out, may be useful.

I think, when it comes to tools, you're thinking of IDEs, etc. We have seen a dramatic rise of those. But most programming teams still have specialized libraries that are developed internally and used from project to project. For example, that one Active Directory security library that is written once and reused from project to project. I think this is more of what the Toolsmith is supposed to do, in terms of
"building specialized programming tools."

For example, in many modern projects I have the arduous task of migrating data from one system to another. This often involves developing a mediator schema or some type of transform. I know the right thing to do is to build a script, just in case I have to repeat the process or add some more data later. But if I'm focusing on Building Shiny New System X, the time I take to write a program to go from Old Piece of Crap Y to X is time taken away from polishing X. That's what a toolsmith would be for.

As for administrators, editors, and secretaries, I'm reminded of Joel Spolsky's work on the Development Abstraction Layer. I think such roles are important, for exactly the reasons he described.

Chief Programmer/co-pilot sounds very much like modern XP Pair Programming, with a very clear Sr/Jr relationship. While TDD/BDD may have them writing their own tests, one can argue that a Tester would still be beneficial for functional testing, and writing al that "glue code" to make testing easier (mock objects, dummy records, etc etc).

I guess my point is, while automation is definitely compelling, don't be so quick to dismiss his ideas. I find myself very surprised that this organizational strategy hasn't been put more to the test.

I would agree that program clerk is a relic of the date of publication, easily replaced by source control.

I find it curious that Brooks' assertions don't seem to have been put to the test in the literature since 1972. I wonder if any case studies have been done regarding this approach?

Also, regarding my assertion that "the assumption that fewer readable lines of code for the same functionality = greater maintainability", I remember studying metrics in Software Project, Process, and Quality Management at Arizona State University...but I don't ever remember running into a heuristic such as this.

Intuitively, it makes sense. A caveat must be made for Perl, of course, which can have one obnoxious line that does everything but is impossible to understand, hence why I included "readable" in the assertion. I wonder if something similar exists in the literature?

If not, it'd be fun to make this Vaidyanathan's Law. (Not to mention deliciously ironic, as I'm all about prolix tautology whereas the law argues for brevity! >:D)

Architecture as defined in Brooks' Aristocracy, Democracy, and Systems Design (The Mythical Man-Month)

"By the architecture of a system, I mean the complete and detailed specification of the user interface...The architect of the system, like the architect of a building, is the user's agent. It is his job to bring professional and technical knowledge to bear in the unalloyed interest of the user, as opposed to the interests of the salesman, the fabricator, etc." (45)

I don't know that I've ever seen a system's architecture defined that way before. It seems limiting to think of architecture solely as the user interface, but the idea of the architect as the user's agent is interesting.

"Architecture must be carefully distinguished from implementation. As Blaauw has said, 'Where architecture tells what happens, implementation tells how it is made to happen.' He gives as a simple example a clock, whose architecture consists of the face, the hands, and the winding knob. When a child has learned this architecture, he can tell time as easily from a wristwatch as from a church tower. The implementation, however, and its realization, describe what goes on inside the case--powering by and any of many mechanisms and accuracy control by any of many." (45)

A great example of the difference between architecture and implementation.