Web Design for Developers

I started reading Web Design for Developers from the Pragmatic Bookshelf on Friday. After a couple of chapters, it seems like what I've come to expect from the Pragmatic Bookshelf (after The Pragmatic Programming and Pragmatic Thinking and Learning): clearly written, insightful, and worth a ponder. Here are my notes:

Chapter 2: The Basics of Site (Re)Design

"It's also important to realize that in the world of design, one person's masterpiece is another person's terrible design."

"Try to understand what your clients think they want from the site you are designing."

"Make sure you understand the real purpose of the site and that you have a feel for the intended audience."

"Once you have the requirements together, start sketching as you discover more. Sketch with paper and pen/pencil"

"Many developers say that clients don't know what they want. I'd say that they just don't know how to tell you."

"You should almost never come to the table with only one design."

"Some requests might not seem all that clear or reasonable. Don't feel overwhelmed when clients say they want the site to look more fun. Having heard this one myself many times, I can only say that you'll accomplish this one by brute force, trial and error, and a little luck. If you accomplish the rest of these requirements, then you'll be in good shape."

"Even worse is when the client asks you to create something exactly like an established site, except different...It might seem like a bad idea at first, but a comment like this is one that you should quietly ignore. Follow good design principles and solicit constant feedback from your clients, and these kinds of requests should work themselves out."

"If you break the requirements into logical steps, they might look like this:

1. Sketch some basic designs and get one approved
2. Select colors
3. Select fonts
4. Implement the basic design in Photoshop
5. Create images for the banner, buttons, and other elements
6. Create an HTML and CSS template
7. Test your designs for compatibility and accessibility"

"To sketch a design, you need to know what the site's layout should contain. What links should be present from the home page? What elements should the home page contain?"

"You've probably noticed that websites tend to have many things in common. Most have a header region that displays the site's name or logo. Many sites also have their main content region divided into columns, and at least one of those columns is often used as a sidebar region that might contain navigation elements or additional information. It's also likely that the site has a navigation bar either across the top of the page of along the left side. Finally, you can usually find a footer region that contains copyright information and maybe some additional links."

"Come up with at least three designs for your clients on every project. Provide a simple, conservative design; a complex design; and a design that aims for the middle, something mostly conservative that also has some splashes of flair."

"When your clients come to you asking for a new site design, get them to do some of the legwork for you. Ask them to identify a few websites they like. Get them to tell you what they like about them."

"I once heard a great presentation from Robert Martin about how writing software was like writing a book. You'll do a first draft and then a few revisions, refactoring until you get it just right, and that's your final draft. Design is kind of like that, except that after you go through all those stages, your client will see it and tell you that he hates it."

Chapter 3: Choosing Colors

"Colors can make or break your application depending on how you use them and blend them together. They evoke emotions and draw attention to important details. This is one of the most important chapters in this book because it helps you build the foundation of a great-looking site."

"You have a lot of things to think about when working with colors. You have to think about the shade of the color, the amount of color, and how the color looks along side other colors. You also need to think about how the color might be interpreted by your audience."

"When people talk about an object's color, they are referring to the hue."

"Saturation is the amount of color in an image. A saturated color is vibrant, whereas a desaturated color looks dull and gray. If you reduce the saturation, you make colors look washed out. In some cases, this is a good thing because it takes the edge off some otherwise harsh or shocking colors."

"Altering the brightness of a color can make the overall appearance of the colors darker or lighter."




"There is a fundamental difference between the way color works on paper or in nature, where light is reflected, and the way color works on a screen, where it is projected. On your screen, the color mixing is additive; in print, it is subtractive."

"When you're working with paints, crayons, and markers, you deal with the primary colors of yellow, blue, and red. You start with all of the colors of light mixed together *white), and filter out what you don't want to get the color you are looking for."

"A banana doesn't actually have any color. It doesn't have any light energy to produce color. Instead, a banana appears to be yellow because it reflects all the light waves that cause us to see yellow, while absorbing all the other waves."

"Computer screens display colors using the additive color system. The primary colors you've grown up with are replaced by red, green, and blue. These colors are mixed together and projected, creating the light."

"The context of a color can greatly influence how it appears in your application. Even if you technically pick the correct colors, you might have to make additional adjustments to make them look right."


"When choosing colors for your application, it's important to think about the various responses your choices might trigger. Using red or blue improperly could trigger an undesired response or could even create confusion."

"Your choice of color influences your users' perspectives, and simply applying a different color scheme to a website complete changes the user experience."

Warm Colors
"As the name suggests, warm colors make you think of warmth, sunlight, and heat. Some people believe you feel warmer if you look at these colors."

"Red is a strong color that can stand for love, joy, happiness, and romance. It can also represent lust, anger, war, an emergency, or danger. Its use in applications is almost always to show a warning or an error message. Red attracts a user's eyes immediately."

"It's hard for a user to focus on yellow, but the color can evoke feelings of intelligence and happiness when used correctly. Many applications use some sort of yellow fade effect to let you know the action you just took was successful."

"Orange can be cheerful like yellow, but it can also be arrogant and superior, depending on the amount of red."

Cool Colors
"Cool colors have a cooling or calming effect on people. They're comforting, and you can use them to tone down a site."

"Blue can be calming, soothing, and cool. It has a tendency to make users relax when it's desaturated. However, as the shades of blue get darker, they can cause feelings of sadness or depression."

"People tend to associate green with nature, hope, health, and responsiveness. However, if used incorrectly, green can also trigger feelings of envy...in addition to envy, green can evoke feelings of greed, guilt, and disorder. Certain shades of green allow the eyes to rest, which can have a soothing effect on your users. The wrong mixture of choices can make your users feel sick or disgusted."

"Purple is one of those odd colors that doesn't appear in nature very often. You might see it in the petals of flowers, but you see it mostly in things that people create. Purple is often associated with royalty and mysticism, mainly because it was extremely difficult to produce in ancient times. Purple is a mixture of red and blue, which means you get some of the attributes of each color. Light purple is often associated with nature, peace, tranquility, and spirituality. Dark purple can evoke feelings of depression. Large amounts of purple can be difficult on the eyes."

Neutral Colors
"Black, white, silver, gray, beige, and brown are unifying colors. They help bridge the gap between cool and warm colors. When used as background colors, they help other colors stand out."

"Black can represent prestige and elegance, and it can be pretty powerful if used in the right context. However, black is also associated with mourning, death, despair, and brooding. When you use black in a design, you must target your audience carefully."

"White evokes feelings of purity and perfection. It's a perfect color for a clean website. Too much white can be boring and sterile, but it makes every other color stand out that much more."

"Brown can stimulate hunger, health, and simplicity. On the other hand, some people observe brown to be a dirty color, and it can evoke feelings of uncleanliness, which is definitely not something you want for your site."

"Beige makes people relax. It is a conservative color that borrows from brown and white. It's a great choice for a background because it can be calming, and will allow other colors to stand out well."

"This color seldom evokes an emotion, but when it does, it's usually associated with feelings of gloom, mourning,

On HTML and structured text as a mechanism to increase comfort of non-technical people with technical content

A writeup on how we could potentially teach document structure along with grammar in english classes and make technical skills more palatable to non-technically oriented people.

The full paper is here.

Sweet Mother of God, Grails is Awesome

A few years ago I did a study on computer architecture education as a preliminary swing on "getting familiar with engineering education research." Being inexperienced and short on time, I collected my data manually (with a giant Excel spreadsheet). Later, when I wanted to analyze the data, I decided to be smart and push the data into a MySQL database so I could query it with SQL. I even wrote an ugly and basic web application in CodeIgniter (back when anything beyond basic HTML forms was intimidating) for professors to update the data.

After the study, I got sidetracked on other projects. The data lived on the server for a long time, until I got a notification that the server was being decommissioned. The most valuable thing for a researcher may well be the data, so I quickly wrote a SQL dump out and put it in my archives.

I recently found myself in a position to extend the original study and use the data again. Being more experienced, I've decided to extend the original schema and build a better application in Grails. The infrastructure building was easy, but secretly I worried.

I knew I would have to write a program to parse the SQL dump, translate the columns, and write in the data...While not conceptually difficult, those types of "data migration" applications are easily prone to errors, may take multiple steps to write...what a pain.

Or so I thought.

Tonight, I found myself testing my Grails 2.0 skeleton by putting fake data in my Bootstrap.groovy and thought: "You know what? I should see if I can load the data from a file in here." Would I lose a whole bunch of time trying to figure out how to translate a SQL dump into domain class instance I could save?

Not with Grails. The awesome, awesome power of Grails.

First, rather than muck with string manipulation or finding some magical SQL-> instance translator, I decided to load up my data into a MySQL instance and then dump it back out to CSV. I probably could've string/regex-searched through the statements, but generally, the less text to process the better, so that worked better for me.

Then I leveraged Glen Smith's excellent OpenCSV. Here, I decided to be tricky. I'd never tried to actually leverage Maven correctly before, but inspired by the example of the Mysql connector

in the Grails 2.0 BuildConfig.groovy, I thought:

"Hey, maybe I don't have to download the JAR and put it in the lib directory. Maybe I can try this whole 'automatic dependency resolution' thing."

So I did the following in my BuildConfig.groovy

repositories {
       inherits true // Whether to inherit repository definitions from plugins
       grailsPlugins()
       grailsHome()
       grailsCentral()

       // uncomment these to enable remote dependency resolution from public Maven repositories
       mavenCentral()
       //mavenLocal()
       //mavenRepo "http://snapshots.repository.codehaus.org"
       //mavenRepo "http://repository.codehaus.org"
       mavenRepo "http://download.java.net/maven/2/"
       //mavenRepo "http://repository.jboss.com/maven2/"
   }
   dependencies {
       // specify dependencies here under either 'build', 'compile', 'runtime', 'test' or 'provided' scopes eg.

       // runtime 'mysql:mysql-connector-java:5.1.16'
       runtime 'net.sf.opencsv:opencsv:2.3'
   }

My tummy bumbled a little bit as my IDE kept yelling at me that my classes were undefined. As much fun as dynamic languages are, there's a comfort to not having those red squiggles telling you you fail at basic namespace resolution...but I resisted the urge to revert to the old school way. I ran the app...Happily, it did in fact download the dependency at runtime!

With the dependency resolved (hopefully), I turned my attention to loading the data in Bootstrap.groovy. Reading the bit on JavaBean binding gave me an idea...but I remained skeptical. Could it be so easy? On first run, the ColumnMappingStrategy didn't work. The Grails error log told me that the 2 of the values of my instance class that I expected to be filled were null. But 1 was not.

class Institution {
   String name
   String carnegieClassification
   String schedule
   static constraints = {
       name blank:false, unique:true
       carnegieClassification nullable:true
       schedule inList:(["semester","quarter"])
   }
}

universityName carnegieClassification schedule
Brown University RU/VH: Research Universities (very high research activity) semester

Ah, the carnegieClassification was binding correctly, but the other two fields were not because of mismatch in the named columns. Would the JavaBean thing fail utterly?

Nope.

I minor edit later...

def createInstitutions() {
       CSVReader reader = new CSVReader(new FileReader("universities.csv"));
       String [] nextLine;
       HeaderColumnNameTranslateMappingStrategy strat = new HeaderColumnNameTranslateMappingStrategy();
       strat.setType(Institution.class);
       strat.setColumnMapping(["universityName":"name", "carnegieClassification":"carnegieClassification", "schedule":"schedule"]);

       CsvToBean csv = new CsvToBean();
       List list = csv.parse(strat, reader);
       list.each {
           def already = Institution.findByName(it.name) ?: it.save(failOnError:true)
       }
   }

Just like that, part of the old data loaded and a roadmap found for the rest of it. No fuss, no muss, in less than 10 minutes. Wow.

Less than 3, Grails. Less than 3.

Has the Software Craftsman movement successfully integrated with introductory software engineering?

By Nicholas Vaidyanathan

Abstract

In this paper, we explore whether undergraduate software engineering curriculums have successfully integrated some of the top advancements in the software industry from 2000-2010 into introductory software engineering courses. Advancements have touched all aspects of the software development lifecycle, including the advent of BDD/TDD, Continuous Integration and social Source Control, and the maturation of design patterns and domain driven design. In order to discover how many of these tools are being leveraged, we examined the offerings of the introductory software engineering course of the top 30 schools according to the US News & World Report and analyzed their course content to determine the penetration of these industrial innovations. We found low utilization of advancements in the field, and propose a roadmap for educators to apply moving forward.

Introduction – The Challenge

Software advances perhaps more quickly than any other field in the world. In the relatively limited time period between 1990 and 2010 alone, the industry saw dramatic shifts that changed the way the world views software. The advent of the World Wide Web, ubiquitous computing via mobile devices, and emerging virtualization have changed what industry partners seek in graduating computer science students. This monumental progression poses a challenge to software engineering educators: how can we properly prepare students to be effective professionals in such a rapidly changing field? How can educators avoid “teaching to the tool” and provide effective generalized knowledge about the software engineering discipline that doesn’t tie their students to the Vendor Lock-in AntiPattern? [1] What are the base set of skills a student needs, and tools a student should be familiar with, in order to be an effective professional in this dynamic environment?

Industrial Attention

Luckily, educators are not alone in having these concerns. Many professional software engineers have also recognized the need for sweeping changes in software engineering education. They have responded to the call by producing a deluge of important “industrial best practice” literature that seeks to educate neophytes. Andy Hunt and Dave Thomas The Pragmatic Programmer[2], Freeman and Freeman’s Head First Design Patterns[3], Fowler’s Patterns of Enterprise Application Architecture[4], Evan’s Domain Driven Design[5], Beck’s Test Driven Development by Example[6], Robert Martin’s Clean Code [7], and numerous others are vital sources that can educate and prepare students. But how effectively is such relatively recent literature being deployed in software engineering courses?

Evolution of Tools and Processes

Documentation of the sheer magnitude of the explosion of tools and processes from 1990-2010 could lead any well-meaning researcher to write thousands of pages and still fail to capture the magnitude of change, so we will full elaboration as future work. We will stick to the highlights.

Languages

The Java Programming Language[8] heralded the rise of an age of interpreted programming languages that sought to achieve platform independence. Extending this pursuit further, dynamic languages such as Python[9], Ruby[10], and others sought to eliminate unnecessary syntax and focus on allowing software engineers to build working systems quickly. These languages are now being used to explore the creation of Domain Specific Languages [11] that would allow engineers to program closer to the intended problem domain. Speed is of the essence, and is one of the hallmarks of Agile software development methodologies[12]. The rise of Agile Software Development as a methodology is well known and well-cited in the literature[], but its attendant ramifications have not been as well explored.

Requirements Specification techniques

The traditional days of SRS specifications have given way to the rise of agile tools such as CRC cards[], use cases/stories[]. Additionally, many tools such as Visual Studio Team Foundation Server 2010[] and JIRA allow business analysts to record requirements in a database driven system. This creates an emphasis on smaller artifacts that can be tied directly to the code that implements them.

Testing and Methodologies

Test Driven Development [] and Behavior Driven Development [] have sought to turn traditional development processes on their head by emphasizing testing from the beginning. Unit and integration testing have become a standard in almost every language with the rise of the xUnit family of tools. Mock Objects[] allow for simulating a live environment without directly having implementation available. BDD encourages leveraging tools like Cucumber[], which enable the creation of executable requirement specifications and end-user centered functionality evaluations.

Design and Analysis innovations

The GoF’s seminal work on Design Patterns[] has resulted in the codification of patterns in a variety of other formats. The enterprise has seen much innovation, including enterprise application architecture[4] and integration[] patterns. Patterns have also been leveraged in language implementation[], and many other areas. Patterns literature has led to evolution of architectural considerations, such as the rise of Domain Driven Design.

Source Control and Configuration Management improvements

While tools like CVS[] and Clearcase[] have offered source control and configuration management options for years, they are quickly giving way to a new breed. Recent innovations such as distributed source control with Mercurial[] and git[], web-based bug tracking and project management with JIRA[] and Lighthouse[], and continuous integration build servers like Hudson[] and Jenkins[] enable shorter development turn-around time, high project management visibility, and automated test and migration of applications between development and production environments.

Are these innovations being integrated into software engineering instruction?

An outline for open source "hack night"/mediated group coding activities

I've recently found myself proposing "hack nights" on multiple occasions in various different group contexts. I've been to a few "hack nights" in community spaces, and have typically ended up frustrated. Generally it either turns into a group of developers sitting with their laptops and working on their personal projects while occasionally trading banter/being lone code cowboys sharing space, or in-depth conversation with one or two people on topics that may or may not be related to coding and little progress on the project.

This is especially irksome, because I know these events could accomplish so much more. I love Free and Open-Source Software. But I also know that the quality assurance work on it is typically awful. Generally we blame and leave it up to the developer to solve problems. To be fair it is their responsibility...but community hack nights could be so much more.

We can use the "hack night" style of meetup as an opportunity to create a learning activity. We can use this meetup to help developers:

• learn about best practices
• learn about new technologies from a more hands-on approach than hearing someone else's "20 minute slideshow on Whiz-Bang.awesome. Go!"
• adjust their workflow and style by integrating with others
• express their own ideas about different architectural concepts and increase community feedback to their "I'm going to rewrite jQuery from scratch because it's a great idea" approach
• make meaningful contributions to the open source community by actually doing all this on REAL SOFTWARE, Not HelloWorld.java or Yet Another Open Source Initiative to Add to the Community.

So I recently proposed a structured event in the Phoenix Grails User Group that I think will meet these challenges, centered around Grails plugins.

I think Grails offers the perfect environment for this type of activity because

• it's modular
• it has automated support for dependency tracking
• it's built in a language and paradigm most people learn in school (Java)

What this means is that most Grails plugins are small, useful pieces. They often feature <10 domain classes [I should do some research to get an exact number], and because Groovy is a compact and expressive language each piece can have few LoC.

Rather than checking out Mozilla Firefox or Apache Tomcat and trying to navigate through a large code base, focusing on small plugins realistically allows "full comprehension of the system in question" in much smaller amount of time. Small, easily digestible systems are easy to digest, test, fix, and modify.

These plugins are typically built from a "functionality" perspective. This means that their test coverage is generally poor, their documentation is typically sparse or "non-enterprise quality" (no class diagrams or Javadocs for public APIs), the code "isn't clean", etc etc. But because they're so small, it doesn't take much effort to "whip them into shape." In theory, even a small team of talented developers could take small applications and dramatically improve their quality in a matter of hours.

But how do you ensure that such an activity is maximally productive? That people learn, good practices are followed, and is a fun event for all?

I propose the following structure:

Preconditions:

Establish a github repository for the group

Select a plugin (preferably prior to the hack session)

Everyone bring their laptops with a local Grails development setup

(perhaps on a special purpose VM. Sugar bear points for someone who rolls out a pre-bundled instance for VMWare Player that everyone can download and get started with)

Activity:

First 20 minutes: everyone individually reads through plugin source, takes notes and makes suggestions

Brainstorming activity (10 minutes): Group derives the intended functional requirements for the plugin by using white board/note cards and making user stories.

• Does the plugin documentation reflect its functionality?
• is there an overview paragraph?
• A set of user stories?
• a class diagram?
• a JavaDoc for API type of stuff [I honestly have no idea how Grails Docs are generated]
• Do the tests fully stress the constraints and methods of each (Should we explore translating the generated user stories into BDD style tests using functional plug/cuke for grails/something else?)
• domain class
• service
• tag lib
• controller
• Does any security sexorcist wanna lead a session on common patterns of attack vectors that grails controllers could be vulnerable to and the types of tests we can write to test their efficacy?
• Are there boundary conditions that the tests are missing? (i.e. things the app should NOT do that are not being tested)

Coding Activity (<=1 hour): write test code to achieve better test coverage and ensure that we don't "break the plugin" during later refactoring

• How do we split this up? Each person takes a card and works on it? Work in pairs? I guess it depends on how many people show up. I'd love to do some XP with an experienced Grails dev myself.

Group Activity Refactoring Analysis (20 minutes):

• Is localization well supported?
• Is there clean separation and proper coordination between the controller layer, service layer, and domain object layer (e.g. is the controller doing heavy processing that should be moved to a service?)
• Are the views too big (are there opportunities to break stuff up into partial templates?)
• Does the domain model "do good things?"
• (for example, I know Joseph showed off an example of doing M:M using two hasMany at the JUG presentation. I am *highly* of the opinion that that sucks, and these should be broken up into other classes via Scott Davis' Tutorial series and the Spring Security PersonRole approach...I am, however, open to persuasion after engaging discussion :)
• is the code "clean"?
• Good variable naming
• short methods
• classes that read like newspapers
• ...other tidbits from Martin's book

Coding Activity (<=1 hour)

• Fix problems identified in analysis. Fix them similar to style in testing phase, and run tests after changes to make sure stuff passes (Sugar points to someone who setups Jenkins for this. I would, but I've never used it. It seems like the "right kind of thing to do")

Extensibility Activity (20 minutes)

• What functional extensions could the plugin accommodate?

Coding Activity(<=40 minutes)

• write BDD functional spec, unit tests, and test driven approach to add features
• add features

Wrapup activity (20 minutes)

• finalize branch changes, send out announcement to grails mailing-list, merge changes into plugin repository and release
• What other plugins may be advantageous to leverage?
• What other plugins may use this one?
• What did we learn?
• What other plugins should we look at?
• iPhone or Android? (HOLY WAR!)
• MOAR Beer.

I surmise that activities such as these may actually be closer to the "missing piece" of the transition between "fresh out of school" and "experienced and effective practitioner." Repeated exposure and practice with activities such as these can potentially give developers a strategy for moving onto bigger applications, which they can then break down into chunks and attack in the same manner. This may actually be important.

For example, when you first start weight lifting, you don't start doing bicep curls with 50 lb dumbbells (unless you're He-Man). Yet this is the equivalent to what many CS new grads face when they're hired by a big company.

"Go do QA/maintenance work on this giant legacy system you have almost no hope of completely understanding. Don't worry, we'll start you with 'small, not important tasks' "

Or, perhaps even worse, they're loosed like cowboys to work on self-contained applications/pieces of functionality in individual or very small team environments. Hence they never gain the opportunity to learn best practices from other experienced people in the team and refine their workflow to be more effective.

But activities like these can serve as a bridge, because they add structure to the programming activity.

Typically, college and other programming courses give theoretical overviews of "concepts necessary", then assign the actual implementation of those concepts as "projects or homework." A requirements spec is given. Then the student programmer, like a painter, is loosed on a blank canvas and told to report back by X with output. Even lab sections in courses typically involve a TA giving an overview of the exercise, then sitting around to answer questions/provide assistance as the artists work on their canvas.

But what if we invert the model and bring it closer to the style of a Reniassance-era painting school? What if we have students reference an already constructed work, copy it, critique it, personalize and extend it? What if we work harder to work through that process, rather than leaving it as a nebulous "homework" phase?

Maybe they'll learn how to be better developers, while simultaneously improving the quality of open source software. I figure it's worth a try.

On Framework Management

Most OSS communities that accept plugins from all contributors run into problems of conflict at some point or another. So many plugins, all doing the same thing...which one should I use? How much time do I as an end developer spend evaluating other people's plugins, before I find issues that cause me to write my own? Without community feedback and active management of the plugin repository, the situation gets quite complicated. Especially if the communities evolve quickly, and no structured review effort is in place to remove legacy artifacts.

I like to analogize the situation to a 12 year old boy who eats a lot everyday, from healthy salads to Twinkies and Mountain Dew, without exercising. He just knows that the food tastes good and he enjoys it. Then, before he knows it, he explodes to 300 lbs. Suddenly doctors are warning him about heart conditions. Worse, in a few years he'll want the attention of the most beautiful girl in school, but may very easily scare her away with his excessive breadth.The boy will be in trouble until he changes his habits and begins to exercise away the accumulation that can literally crush him under all its weight.

Similarly, even useful frameworks can traumatize developers with features that promise easy use but then fail to provide the tools necessary (documentation, functionality, intuitiveness) to make good on those promises. For example, take most CMS systems. They can sometimes give wonderful "quick-win" solutions for simple tasks, but often lose favor with developers who have to do anything "more complicated."

The problem is, most OSS communities are focused on pushing new features and functionality. They are also by nature inclusive. Normatively, one can argue that the end goal should not be to erect significant barriers to contribution, as that goes against the spirit of the movement.

At the same time, without active review of contributions and acceptance of good ones, rejection of bad ones, and elimination of old ones that are super-ceded by "new and improved" ones, "plug-ins" become very analogous to what happens in the human body when cancer afflicts. Cancer is, biologically, unrestrained growth. A condition that occurs when cells "refuse to die" as a result of mutation. Just as the system that is the human body is destroyed by this unchecked process (a "biological memory leak"), so too can such processes dramatically affect the usability and adoption of OSS systems.

The solutions in the biological case for the maladies of unchecked growth are: change in process, actions (fat kid= eat less, exercise more) and introduction of external agents to eliminate problematic elements (cancer patient= chemotherapy and radiation). One can make the case that many OSS projects are already or becoming "fat." While it's probably too early to say there are any "cancerous" OSS projects, the potentialities are already there.

The problem is fundamentally one of governance. Abstractly, one could consider it to be the same problem the United States Federal Government has today: unlimited feature creep leads to unchecked expansion of the system base. Soon the system becomes so hefty that velocity of new development slows down or each new feature has massive unintended consequences.

This is the same lesson Windows has been learning for years ("My system doesn't work! Windows sucks! [Actually, it's because I'm using software X with hardware Y in configuration Z and one of the drivers was incorrectly written.]" Now other systems are experiencing it as well, such as Drupal 7 which had dozens of modules that claimed they "would be ready on launch day" but were still in Alpha/Beta stages, and which regularly have bugs reported such as "Drupal 7 doesn't work because of ___ [broken module]"

Resolution is seemingly impossible for the problem of unlimited feature creep. It is, after all, human nature according to Economics (see: scarcity). Mitigation comes from recognizing the problem and creating an according infrastructure to address it. We already have an example for this, if we once again turn to the human body.

The human body does not only contain red blood cells, or have organs for the sole purpose of "achieving functionality" (arms, legs, etc). It also has white blood cells, and organs like the liver whose duties include the elimination of impurities. There is an immune system in place that actively handles maintaining the health of the body. Most software systems have no such components. We consider a system "healthy" if it doesn't have a lot of outstanding bugs and meets most user needs. But such a view can be analogized to surmising that because a person doesn't have a disease, they are healthy. Absence/minimization of maladies can be one useful indicator, but as many in the medical industry are coming to believe cannot by itself give the "whole medical picture."

In small software systems, as in children, an under-developed immune system results in manageable fragility. Healthy children have just enough immunity not to perish from pathogens and develop resistance with respect to time. It can be expected that small software systems can mimic that cycle of maturation through build->test->release->patch cycles. If the "immune system" of the software develops throughout its maturation cycle, i.e.) as new features are added, active refactoring eliminates waste and inefficiency, regression testing verifies system health, etc, the system can be said to be healthy and likely resilient to a variety of maladies.

Larger software systems face a different set of constraints. Frameworks for application development, for example, can be analogized to natural ecosystems. In natural ecosystems, there are predator/prey relationships. These relationships have to be in harmony in order for the system to remain sustainable. Software frameworks often advertise themselves as "ecosystems"...but what are the predator/prey relationships? Are other frameworks "predators?" Only in higher-order ecosystems.

Within a framework itself, the "modules" or "plugins" must comprise the "predators/prey." They compete for community mindshare and usage, as such they sometimes conflict and sometimes work complimentarily. In order for the overall system to remain healthy, sometimes new modules must eat old modules or old modules must consume specialized features offered by new modules. This consumption can only happen if modules are actively removed from the "availability pool." Instead, many OSS frameworks currently function like deserts with no vultures. The rotting remains of ancient and broken modules litter the landscape. Many times it is up to the module developers themselves to clean up their mess, which can be analogous to asking a corpse to dispose of itself.

Whereas nature has itself created a "governance" whereby specialized organisms exist to "remove the dross"/"recycle old elements", software frameworks have no natural mechanisms. We have to "write our own garbage collection." Consequently, it is of fundamental importance for framework architects to take such considerations into account throughout the evolution of a framework, lest the system becomes ultimately unstable and breaks down. If nothing else, policy must be established to determine how the ecosystem will be managed.

There are a variety of approaches available to handle stablization of the ecosystem. Some automated, some not. For example, we could imagine a tool that periodically purges old modules based on a variety of metrics. Perhaps if a module is released as a "beta" for more than 1 year and has no activity in the form of source control repository updates or releases, it should be removed. When other users "fork" or create sub modules that leverage that module, which themselves have activity, the old module's activity log can be updated (hence, the module is active, in that its lineage is). This requires the ability to store relationships between modules, which most centralized repository tools such as Apache Maven already have.

Once we have this ability, we can do more. We can tag "active" modules with functionality descriptors which allow us to scope out the "range" of the module. Simple tagging can allow us to create "feature matrices" whereby modules can be queried, such as "give me all the modules that can draw 3D bar charts using Javascript." Grouping modules of related functionality allows for comparison between the modules. For example, Grails has developed a very basic tagging and user rating system where the community can participate and rate modules. This works for a simple metric. It is of course fairly subjective, and gives no insight into the actual implementations of the modules, only the perceptions of their functionality from end users. A more advanced approach would use that rating as part of a metric that combines other factors. For example, basic test applications can be written as "benchmarks" for the modules. These benchmarks can be used to compare the functional efficiacy/equivalence of modules (e.g. For this given Animal domain object, generate a PDF. Assert that each module can do it, and that the output files all contain the animal's name) and their relative performance (in terms of processing time, memory allocation, and other low level metrics). These benchmarks can even integrate code analysis tools to evaluate the modules and produce comparisons on LoC, convention issues, potential bugs/memory leaks, etc.

Benchmark analyses, in conjunction with user ratings and other metrics, allow us to define a common "domain" for the grouping of modules. Combined with the range, we can develop axes for evaluation and comparison. Plotting these modules in the community derived "suitability space" would enable framework architects and module developers to graphically compare modules. This can lead to pooling of efforts, elimination of unfit modules, and even refactoring of large modules into sub-modules. It can also provide clues as to what "third party components" should be investigated for integration into the framework core. If certain modules are so widely adopted they're almost considered part of the core, they can be integrated. Alternately, the core can remain relatively stable and small but "flavors" of the framework with popular modules (that encapsulate a set of pre-defined common functionality requirements) can be developed.

Such use cases, automated pruning of dead modules and benchmarking of live ones, lend themselves to automation. Such automation, however, still requires manual involvement. The metrics for pruning must be established and benchmarks must be developed by the community. This leads to a greater point. It's not enough for framework architects to focus solely on extension of framework capabilities and elimination of bugs and issues within it. Just as software projects require testing, refactoring, and overall quality assurance staff, so do software frameworks. Perhaps on a larger scale. Most frameworks up to this point have taken the approach of a human body with all red blood cells, and no white blood cells. They grow in scope and size without dedicated teams that seek to contextualize that change and modify the architecture accordingly.

For example, ASP.NET was eager to add lambdas and LINQ in revision 4. Yet the ASP.NET web application component still offers almost a myriad of ways to get the base path of a server, with very little guidance as to the merits/demerits of each or even investigation as to whether some of those should be refactored/removed. This suggests that it has not been a priority within the framework architect's enclave to maintain architectural unity and strive for simplicity. These are fundamental requirements espoused by Brooks in The Mythical Man Month for successful software endeavors, and Booch has also explored the need to combat increasing entropy and inertia by concentrated effort in his series, On Architecture. .NET is certainly not the only example, of course. Some have taken to calling the Java Programming Language an "everything and the kitchen sink" endeavor, which some hypothesize delayed the release of Java 7.

It's not uncommon for any widely used and successful software framework to run into these types of issues. Successful mitigation strategies are rare. Mainly because scale was "unplanned for", and no established quality assurance planning was integrated in the beginning.

Successful framework architects must establish a "governing board" of minds who oversee and debate the evolution of the framework. Part of that board must include a review board that actively manages the framework ecosystem. In fact, one can argue that as a framework grows in user base and module scope, the amount of effort to maintain the ecosystem should begin to parallel and in some ways even outstrip the efforts to improve the core. This quality assurance involves more than just filing bug reports on core features and write test plans. Rather, these users should help develop the benchmarks, taxonomize the overall ecosystem, and provide guidance in the form of documentation and refactoring support to module developers within that ecosystem. For example, within the Java framework matrix multiplication is offered by many modules. A "mathematics functionality coordinator" could contact the lead developers of the different modules and work with them to establish benchmarks to compare them against, as well as common interfaces that would allow for intercommunication between that piece of functionality with others. Java has already done this successfully with the Java Persistence API.

Framework evolution, especially in the Agile age, is a chaotic and rapidly changing process. This is part of the joy of software. But that joy is not without its hidden pitfalls. The point is not to slow down the rate of progress to avoid those pitfalls, but rather understand that energy must be expended to overcome them through active management of ecosystem. Frameworks need to lose some fat too, lest they find their lunch eaten by smaller, quicker frameworks that gain more mindshare for their simplicity.

Grails PDF output: PDF versurs Rendering plugins

I recently got a requirement where I needed to output a GSP as a PDF. This is not uncommon, and there are a couple of Grails plugins available for this. The most popular are the PDF plugin and the Rendering plugin.

This is exactly the type of thing that happens a lot in open source communities. There are often many plugins which seem to do the same thing. As a poor end "user" (a developer relying on someone else's code), pretty much the only thing you can do is download it and try it out.

I decided to try the PDF plugin. For one, it has better ratings in the "plugin rating system" of Grails. Plus, the original version was written by Glen A Smith, who's a co-author of my favorite Grails book, Grails in Action. Surely, that would be good, right?

...Here I am, 4 hours later. After twiddling with PDF for hours, I just installed and tried the Rendering plugin. Got out of the box functionality in 10 minutes. Ouch.

I had a fairly simple use case where I wanted to render the results of a partial template as a PDF. I struggled along the learning curve of the PDF plugin. I was not very impressed with the its documentation (or the fact that it polluted my project with its views as a way of providing documentation), and after figuring out different things (Did the pdfForm tag accept views in other directories, or did they need to be in the pdf/ directory? How do I pass parameters to my template? The simple case of passing primitives is covered by the demos, but what if I want to pass a domain object with pre-populated values that the template is supposed to reference?) I got stuck on it with some Null pointer issues generated by the plugin like so

I consequently tried the Rendering plugin, and got that working quickly.

Looking at the specs and code of the PDF plugin, it's not surprising. It's been around since Grails 0.6, and its code under the covers doesn't seem to be as modern or flexible as the Rendering plugin. I could be wrong, but it seems like the Rendering plugin has more community support and functionality moving forward, while still managing to get the special purpose job of PDF generation done pretty well.

For convenience, I created a sample project with both to showcase the problems I ran into. It is available here. Perhaps they stemmed from ignorance. I invite others to give it a try and respond with comments.