The Passion of the Developers

A thousand little decisions go into the creation of a great product. Most are not in the specs or master plans. Instead, they are implemented by testers and programmers. Yet, these are the same people that are traditionally lampooned as geeky and out of touch with the target customers.

What would happen if the developers possessed a deep understanding of their customers needs and desires? Suddenly, those thousand little decisions aren’t introducing random noise into the product. Instead, they are pushing the product forward. A single decision might not make much of a difference, but taken en mass, the momentum generated by enlightened developers can result in a massive reduction in design risk. You’ll develop the right product sooner and with fewer errors.

You can’t unfortunately just say “Dude, I looove my customers” and expect to get results. To harness the power of the passionate developer, you need the following pieces in place:

• Empowered developers
• Meaningful customer feedback

Empowered Developers
One philosophy is that we need better specs and those damned monkeys need to implement the specs exactly as designed. Better command and control system and more rigorous process is obviously the trick to success. This tends to generate poor results.

Imagine that you are a highly creative individual whose life is reduced to a series of highly detailed checkboxes that you are told to fill out one after another, day after day. If you deviate, you are punished and ridiculed. The results are gun-shy, non-communicative production workers that rely on obfuscation and deliberate bending of rules in order to express their innate creativity. Many wonder why they got into this dreary business in the first place. Others just give up.

When indentured coders and testers make design decisions, they lack both the experience and the desire to do the right thing for the user. They’ve been trained to take orders from management instead of building up an intimate empathy with their user. In the best of worlds, they implement uninspired versions of the features described in the specs.

In the worst of situation, they labor in secrecy, adding a little bit of their own uninformed creativity back into the product. When confronted with the havoc wreaked by Byzantine architectures and convoluted, unnecessary features, the humanoid production resources are surprisingly defensive. How can you be angry? They were just trying to salvage a small bit of their creative souls.

Such feature creep is not the fault of the individual testers or programmers. It is the fault of the team and management for turning a creative and invigorating process into menial labor where talented creators have no other option but to rebel.

We need a different philosophy based on passionate, educated team members that put the user at the center of their world. Developers gain the power to make their own decisions through their deep understanding of their customers.

Simple beliefs
Empowerment is a word that has been stripped of much of its initial meaning by the hippy liberals of the world, but the concept is rather simple. An empowered person is someone who has the ability to make their own decisions.

Let’s look at some building blocks that need to be in place before you can have empowered developers. You need to believe these are true and the people on the team need to believe that they are true. If these foundational beliefs are weak, it is very likely that you’ll end up falling back on command and control decision making structures in the face of emergencies or pressure.

• Developers are passionate people: Even technical people burn inside with an inner flame of intense creativity. It can be hidden. It can be misdirected. But you must believe that it exists. The other insight here is that developers are motivated by very human issues. Dream, fears, respect and social hierarchy have a greater impact on productivity than economic incentives or logical rules.
• Most decision should occur at the edges: Management serves a useful role in maintaining group cohesion. They rarely however, have intimate knowledge of the production issues experienced by individual workers. When developers at the edge with customer empathy are encouraged to solve problems that they happen across, a hundred issues that are invisible to managers start going away. A well functioning team is bursting with these small acts of ‘goodness’. Over the course of a project, the momentum results in an order of magnitude improvement in product quality. The most important outcome of this philosophy is that the people who build a piece of customer work flow hold absolute responsibility for it’s usefulness to the customer. “I’m just doing what I was told to do” is the vilest of attitudes.
• Encourage learning through failure and feedback: The urge to control production often stems from fear of others making mistakes. “The morons are going to just screw things up!” is a common refrain. If you remove the opportunity for people to make bad decisions, you slide rapidly down the slippery path of command and control. An empowered developer has permission to fail. In fact, rapid, low cost failure is actively encouraged since it is the single best way to learn about how to make the product better.

Customer feedback
Now we have a team of passionate people. They are solving problems at the edges and they have permission to fail. Wonderful! We still, however, are missing the most critical element. We need to build a strong customer feedback mechanisms that encourages the team to create a product that matters.

• Identify the customer
• Believe in the customer
• Get close to the customer

Identify the customer
Identifying your customer is where an amazing number of teams fail. They become focused on a bit of technology and imagine that users will just appear out of the woodwork. The whole process of iterative product design requires an identified customer so that you can know when you are on the right track and when you need to change direction.

The first step is to write down your customer definition. You’ll take into account market data, economics, opportunities that you’ve noticed and a billion practical details. It will feel like a very intellectual pursuit. The next step is to go out and find an actual, breathing human being who represents your customer. Customers are not data points. They are people that you can question, observe and befriend. The customer definition is just a tool that helps you narrow down your search. The real win is when you can get your customer in the room with your developers and they can drink a beer together. PBR FTW.

This isn’t easy. You may not be able to find a single person who represents your typical customer. You may end up with multiple customers that represent different needs. You may find that people on the team had very different ideas of who the customer might be. You’ll likely redefine your customer multiple times once you start talking with your initial customer prospects.

It is okay if it is hard. Don’t give up. Choosing your customer is the single biggest product design decision that you will make over the course of your entire product’s development. My belief is that no project should be green lighted for even minor development unless the team has had extensive conversations with an identified target customer.

Belief in the customer
The user of your product must be worthy of great passion. Remember, we are dealing with passionate developers thrive on living meaningful, creative lives. When the target customer is considered unworthy or boring, the development team will feel little urge to empathize or go the extra mile.

Product development is a service to others. The products we produce make someone’s world a better place. The team must be able to explicitly articulate why it is worth spending years of their precious lives providing the target audience with a new bauble. If you can get everyone on the team to passionately proclaim that they wish to serve the customer, you have built a powerful unifying vision that will carry the team through hard times.

It is worth noting that practically any customer can be worthy of great passion. All customers are people with dreams and passions of their own. They have tricky problems that result in very emotional human costs. An important aspect of connecting the passion of the developers with the passion of the customer is to identify the human, emotional aspects of the customer problem and bring those to the attention of the team.

Get close to the customer
Once you’ve identified your customer, you need to figure out ways of gathering information about them. One of the great Zen mysteries of product design is that in order to successfully serve your customer is that you should never give them what they request. So you can’t just grab a customer of the street and ask him to design your product. An average customer is good at identifying problems, but they are poor at prioritizing, understanding larger patterns and generating elegant solutions. That is your job. Your customer is there to let you know when you get things right and when you get things wrong.

The feedback cycle is pretty straight forward

• Identify a problem or opportunity
• Build a valuable solution that fits the available resources constraints.
• Present the solution to the customer and record their response. Do they realize the value of your solution?
• Use the data to identify new problems and opportunities. Look at the big picture and start the cycle all over again.

There are a plethora of good techniques available for getting closer to the customer. IDEO has their 51 ideation cards, college design courses teach books worth. They operate at different timescales and give you different types of information. Implement multiple ones and see which ones work the best for your particular customers. Here are some of my favorites:

• Use your own product: Try to replicate what you see your users doing. Mimic the masters so that you understand their thought process by doing. This builds developer empathy with your customers and is one of quickest feedback cycles around. It also however, can be highly biased.
• Onsite customers: Make a customer or two part of the development team. You get very rapid feedback and lots of cultural transfer into the team. The downside is that customer objectivity can be polluted by constant contact with the team.
• Observe customers using your product: Set up usability studies and gather both quantitative and qualitative data. Every team should do this, but it can be expensive and feedback only happens in spurts.
• Hire psychologists and ethnologists to study your customers: You’ll often gather key insights that would otherwise slip by someone not trained in methodical observation. Making these results actionable often takes additional work.
• Listen to lead users: Some users are actively solving the same big problems that you are attempting to solve, often years ahead of the market. Find them and learn from their successes and failures.
  

You are always balancing three different aspects of the customer feedback.

• Reduce feedback cycle time: The more quickly you can fail, the less effort you can put into each failure and the sooner your can correct your course
• Improve objectivity of results: It is easy to get biased results. Many of the issues dealing with customer psychology are fuzzy in the best of situations and the developer’s own interpretations often dramatically skew the data.
• Improve clarity of results: With more data comes more noise. Boiling down the flood of information into actionable, meaningful results that the team can understand is an ongoing activity. Many teams collect customer information, put it in a binder and then never look at it again. If you don’t act upon the customer feedback, it isn’t useful.

There is a substantial cost to all of these activities. You’ll need to change your development practices. Daily builds and regular check-ins help ensure that you always have a product that your can put in front of users. You’ll need to spend money. Bringing in users, ethnologists and usability experts adds to your headcount. This is the cost of doing business.

Barriers
I’ve laid out two pieces of a successful product design team. The first is empowered passionate developers and the second great user feedback. Without the first, user feedback is either ignored or fails to permeate the multitude of small decisions that go into building the product. Without the second, the team spirals off into wasteful directionless exploration.

The single biggest barrier to creating a team of developers that are passionate about their customers is the core cultural beliefs of the people already on your team. Chances are good that they rose to where they are today by mastering a traditional command and control style system. They’ve been personally rewarded throughout the years by checking off the check list or writing checklist for others. They got a big fat raise for staying late and adding those 12 new features from the spec. When the product fails, they are told, “We should have added those 6 more features that got cut because developers were so slow.” The weak ones leave and the ones that are great at silently adding questionable features are promoted and promoted again. They are trained, their egos are invested. After five to 10 years of this, an experienced developer builds up an unshakable, unquestioned belief in command and control style development that only falters when they finally die off.

A thousand little successes do not start with a mandated policy. They begin to gain momentum when the role models within the company start acting upon the beliefs in concrete ways. This is where most process change breaks down. People say one thing, but revert back to habitual behavior when the going gets tough. They still believe, deep down, that the old method of command and control is less risky than trusting a team driven by passion and customer feedback.

Only when people on the team have personally experienced success using the new techniques and they’ve adopted new beliefs as a core of their personal philosophy will momentum sweep through the entire team. Several items need to be in place:

• The right people: You need a core group of people willing to believe in both developers and customers. These folks are rare since the current system isn’t focused on producing them. If you find one, hire him or her immediately. The culture they bring to the team is worth as much as their skills alone.
• The right situation: Often people need to be in the middle of a failing project before they will consider that there might be alternatives.
• Time: Smart, strong people do not give up their culture overnight. After years of steady and obvious success, they’ll start to shift. Start with a like-minded core and grow it slowly.

Very few companies have all these items align. I was chatting with one insanely successful game company that follows many of these methods. They’ve given talks on their process. They’ve partnered with other companies and attempted to teach their process. Each time it fails. Imagine a company that consistently releases highly profitable AAA titles merrily giving you their secret formula. Would you take it?

Yet no one does. It would require companies giving up their most fundamental beliefs about how software is designed and built. They would need to give control to the lowest people in the organization, embrace a constant stream of failure and put customers, not their creative urges, at the center of the design process. That, my friend, is scary. Product success is cool, but a shot a being Top Monkey in the current ineffective system is much more attractive.

Why are we in this business again? Where is our urge to build great and wonderful things that serve the world? It seems a shame to let tradition, habit and hierarchy get in the way.

Conclusion
Lately I’ve been contemplating a nuclear bomb of a statistic. 80% of software features are either never used or used very rarely by customers. Anyone who does software development should be staring at that number in shock. 80% of your sweat and tears has little to no customer value. In most cases, customers would be just as happy if you were to give them 1/5^th of the existing product and make that little sliver work better. How much time and resources could you save if you only delivered the right features?

This in turn has gotten me thinking about the common software development practices, the ones that fixate on execution risk instead of design risk. What we build is just as important as the fact that we can crap out a product by Christmas.

Passionate developers that empathize and listen to feedback from customers form the foundation of an efficient team for one central reason: They are very good at reducing the risk of building the wrong product.

• Team members more readily identify areas of design risk that require iteration with customers.
• Any big design mistakes are caught early through customer feedback.
• Small decisions are made more rapidly with fewer errors.
  

You aren’t going to save 80% of your development time since all this iterating does chew up a serious effort. But you may save 50% or 60% of the budget. More importantly, you may end up releasing a product that is of value to someone that you care deeply about.

Passionately yours,
Danc.

Read more!

Design Testing: The use of addiction metrics to force rapid evolution of innovative game designs

One of my goals with this blog is to formulate a 'new game development methodology' that empowers the little guy and helps the growth of innovation in the game industry. How do we build innovative, highly addictive games more quickly and with lower risk? Part of the answer is the rigid application of gaming metrics to the process of improving player addiction.


The Legacy of Cowboy Designers
The traditional designer is a cowboy designer. Modern game designs are the result of the messy, content dependent process a cowboy designer intuitively follows when building a game.

Cowboy Programmers
The term comes from the land of programming where early programmers would whip out l33t code in as little time as possible. Cowboy programmers were lone guns, experts in their field who possessed a deeply intuitive understanding of what works and what doesn't.

Coding standard, methodologies, even team work were taboo for the ancient cowboy programmers. Their code was inscrutable and many decisions seemed arbitrary. Troubles inevitably arose as the industry matured. Project didn't scale and many failed as they were bogged down in a plague of bugs. Eventually the world figured out better ways of programming that were more reliable, less risky, and produced better results. God bless process advancement.

Cowboy Designers: Copy cats with a hip attitude
Cowboy Designers are similar in many ways. They shoot from the hip when it comes to decisions, relying on their own finely tuned sense of 'fun' to design systems and create requirement docs. This sense of 'fun' is typically built up after internalizing the game play of dozens of similar game titles.

Such expertise works well when you are creating a clone or focusing on the later layers of the game design where you can't do much damage. Adding the 101st 'designer inspired' Pokemon is about as risk free as adding the 100th one. Subtle, oh so well crafted variations on existing themes are the bread and butter of a cowboy designer. The 'I could build it better' syndrome that drives many game designers is not only a contributing factor to the stagnation of innovation, but is actively encouraged by most game publishers as a means of reducing risk.

Cowboy designers stifle innovation
The big problem is that intuitive cowboy designers have high failure rates when it comes to inventing core game mechanics. Experience in pre-existing genres is a poor guide for success when your job is to put together new rules that result in dynamically different psychological scenarios. When cowboy designers attempt to refine a new genre, one of two things typically occur.

• The half-breed design: The designer mixes two well defined genres. Since their decisions are informed by experiance and not psychology, the result is rarely enjoyable to play.
• The mush design: The design mixes multiple genres together with rules that are untested and arbitrary in nature. Randomly designed games tend to a remarkably low success rate.

Either way, the result is ruined teams and failed games. It is no wonder that publishers are adverse to large investments in innovation.

Bad process, not bad people
It doesn't have to be this way. We are simply using the wrong design methodology to build innovative games. The old cowboy method only works with Shooter Clone #64. It fails miserably when attempting something new. With the right design methodology built around the concept of make risky game design decisions painless, innovative games have the opportunity to prosper.


Introducing feedback: A miracle design tool
What we really need is a reliable feedback mechanism that lets us reduce investment risk in order to create a safety net for innovation.

The modern feedback desert
Consider the traditional feedback cycle in game design. You spend 12 to 18 months building a game. You recieve the majority of your feedback from traditional game testers and internal 'team testing'. The information is very useful, but runs into several difficulties

• The information is subjective: Most feedback is qualitative and is filtered through a pre-biased team of hardcore game developers.
• Feedback is not statistically valid: Testing occurs with small numbers of testers that do not accurately represent the target market, nor are their opinions verifiably the same as larger market.

At this stage you still don't get a chance to react. Almost all gameplay fixes occur in the higher layers of the game design due to its lower risk nature. You can change some art or a few engine variables, but rarely is there time to alter core game mechanics due to the exponential cost of change in a content heavy system.

Once the game is released and in the public's hands, you finally get your first pieces of accurate feedback on your design decisions. You either sell a lot of copies, or you don't. If your game happens to end up a failure, there is no second chance. You didn't get it right the first time and now your entire team will be culled in a grand bloodletting by your disappointed publisher.

This isn't a healthy feedback cycle. The opportunity to make meaninful changes is limited from an early stage. Those who make mistakes are punished dramatically. Those who survive see their lifeless brethren on the roadside and learn that risk taking is dangerous.

Specifying a useful feedback mechanism
We need a tool that:

• Rapidly informs us when design decisions unbalance the game
• Lets us test multiple variations on a rule without risk
• Allows us to see the effects of a change before we invest heavily in expensive, difficult-to-change content.

Such a design tool would allow incremental investments in new game designs. If you make a mistake, you can back that change out without putting the entire project at risk. Since the cycle time on between changes, feedback and exception or reject of the change is short, the team can iterate through a series of changes quickly.


Enter the Metrics
There are a wide variety of testing systems available that give us interesting feedback.

• Unit testing
• Market testing
• Design testing

I'll briefly describe each the first two and then explain how Design testing can radically change how you go about game design.

Unit testing
The most common testing in game development is the unit test, borrowed from Agile programming methodologies. These are covered extensively in a wide variety of books and websites and deal primarily with code integrity and refactorability. This is certainly good important stuff that is essential to creating an agile game design methodology. However, unit tests address only programmer risk, not design risk.

Market testing (aka Market Research)
Another common method of product testing involves giving a product sample to users and having them rate how likely they would be to purchase. Market tesing is a huge field and contains everything from focus groups, to concept testing, to full on market testing with a wide scale deployment of the finished product.

Traditional market testing has some fundamental problems when applied directly to game design.

• Expensive: This restricts its use to only the biggest of game developers and publishers.
• Provides limited insight: Second, and most damning is that it is nearly impossible to tell anything about the addictive qualities of a game without actually playing it. I can show you a box with a guy with a gun on it and ask potential players if they would buy it. But such a survey gives me no meaningful information on whether or not I have the next Halo or Daikatana on my hands. "How does it play?" is critical competitive information.

Games, as a testable product, exist in a market research vacuum. Many of the tradition techniques honed over decade of consumer product research simply do not apply. They don't capture 'addiction', the competitive essence of games.

Design testing
We need tests and metrics that capture such ephemeral qualities as 'fun' and 'addiction'.

What makes me think we can test 'fun' and 'addiction'? I believe that core game mechanics rely on relatively simple psychological reward schedules. A successfully addicted player exhibits easily identifiable behavioral symptoms. By tracking these symptoms in a statistically valid manner, the designer gains useful feedback on the addictive properties of their gaming system.


Common Metrics for Design Testing
Testing for addiction is easier than you might imagine. The following are easily gathered metrics for measuring system-wide addictive behavior.

• Length of playtime
• Intensity of play time
• Willingness to play again
• Length between play times
• Number of play times
• Spot exit surveys

Game Token Metrics
You can also get more atomic and measure metrics for each game token in order to dig down into why a particular pattern of addiction is occuring.

• Use time of each token
• Frequency of use for each token
• Gap between token use
• Spot surveys of user's token enjoyment

ROI metrics
Finally, you can gather ROI metrics by combining the information from metrics above with cost of production information from your project tracking. This gives you some interesting information on where your development investments are paying off.

• ROI of each token: Calculated by use time / development cost.
• ROI of each game system

Intriguing results immediately pop to the surface once you implement ROI metrics. Additional level design has a marginal ROI. Character art is the same. You can add a new monster or a new level to the game and the addictive qualities of the title don't budge an inch. On the other hand, add a new powerup system and watch the addiction rise.

Control Charts
You can track these metrics on control charts. This simple charting method tracks changes in specific metrics over time. When a system is changed, you can usually see the results immediately in the control chart.

In general there will be one or two key metrics that (Key Performance Indicators, or KPI) that give you a strong indication of the addiction of your gaming system. Other metrics will be secondary factors that influence you KPI. For example, the reuse time of the powerup system is not the single most important factor in the game, but it influences total session playing time, which is your primary indicator of addiction.

Using the data
Once you have the control charts populated with data, it is a simple matter following a clearly defined change regimen

1. Create a design change
2. propagate that change your game players
3. Measure the results
4. If the change is positive compared to the previous baseline, keep the functionality.
5. If the change is negative or mixed, create a new set of changes
6. Track the key metrics over time to ensure that there is a steady improvement.

Other areas of future exploration
This is a very rough overview of the techniques involved in design testing. It is both a broad and deep field that borrows from many well-developed ideas in the world of market research and process engineering and applies them them to the problem of game design. Other topics include:
Batch testing: Test a wide number of variations in game design mechanics simultaneously. Take the best results and explore them further.
Tie KPI to financially meaningful results: Use regression analysis to link key statistics to financially meaningful results. For online games, measure re-up rate on subscriptions. For ad-based games, measure customer referral rates and number of impressions. For shareware games, measure initial purchase rates.


Design Testing Limitations
Design testing is the core of a rather radical new game design methodology. Let's take a look at some of the limitations.

Not every game can be design tested
Design testing is not for every team nor is it for every type of game. To borrow a term from the agile programmer world, most modern game designs are poorly refactored. They are clumsy, non object-oriented messes of content spaghetti strung together by cowboy designers and their complacent teams of artists and programmers. The typical modern game design has the following attributes:

• Change is expensive.
• Testing takes forever.
• Development cycles are long
• Static content is king

None of this is conducive to an effective design testable system. I think of applying design testing to an adventure game and shudder. The sheer mass of the static level content combined with the linear sequencing of content results in a system where a change to one location has no effect on any other location. Players are likely to play the game only once and it will take them 40 hours to complete. Good luck getting any timely feedback.

Requirements for a design-testable game
To use design testing as part of our process, we need a game design that is ammendable to thorough application of the technique. The following are some key characteristics of a design-testable game.

• Refactored Design: The game is composed of highly reusable object-oriented elements. Changes to these elements propagate throughout the entire game system.
• Game Mechanics focused, not Content focused: Static content in the form of level designs, sequenced boss attacks, fixed plot points, etc is rarely used. Instead the focus is on interesting game rules, meta game rules, dynamically generated levels to create an enjoyable game experience.
• Automated update mechanism: The game designer can rapidly push changes out to a population of game players.
• Real-time metrics: When a change is made, statistics on current player usage are immediately sent back to a central electronic dashboard. Most commonly this will be through an internet-based tracking system connected directly into the game.
• Large population of game players: Statistics are worse than meaningless if you do not have a pertinent population to survey. Testable games require a large standing population of active game players. This suggests extensive open betas and other mechanisms to encourage player interaction before the game is finished. Subscription-based models also work well with this requirement.

Markets ripe for design testing
Online games have a clear advantage. Many of the tracking systems are already built into the web and you already have logs and a database ready to receive the data. You are guaranteed 100% correct information since you see everything that occurs. MMOGs are already doing many of the things outlined in this essay. Their success is readily apparent and I challenge you to find a more addictive genre.

Consoles are moving online in the next generation and most gaming PC's are online already. The technological infrastructure is certainly possible. All it takes are teams innovative enough to improve their development process. Indy games, Nintendo DS titles, and mass market consumer titles all are place where new methodologies might blossom.


Is design testing worth it?
If you want to make a design testable game, you need to throw out decades of highly polished game design experience and theory. You need to rely on cold metrics instead of your warm fuzzy 'I could do it better' cowboy designer instincts. You need to shun common content heavy genres that you grew up with and love in the deepest core of your gamer heart.

What you lose
Design testing fundamentally changes how games are developed.

• Long development schedules cloistered away from the public: Feedback critical to the product's final success. Alphas, Closed beta, and Open betas become essential tools to releasing a polished game.
• Offline games: If you aren't online, you've got no way of gathering data about gameplay.
• Static level design: You need a refactored game design that allows changes to be made quickly.
• Plot: If the ROI isn't there, kill it. You've got data that proves you better things to spend you development time working on.

What you gain
What you get in return is the ability to make radically addictive, highly competitive games with limited risk of failure.

1. Increase your competitive advantage: The other guy is spending all his effort just to maintain his position at the top of the king-of-the-genre battle. He invest in mature genres and every game burns out his hardcore audience a little more. You can come onto the scene with a fresh new title that is more addictive than his current offering. When his FPS is merely one of many such competing titles, your title is a one of the kind 'must have' title.
2. Reduce your costs: Instead of spending millions on movie level content, you gain your addictive rush from intelligent, informed game mechanics. The result is a lower cost structure. Because you have ROI metrics built into your design
3. Reduce product failure risk: From the very beginning of development you know, to the decimal point, how addictive your game is with your target market. This lets you cull the bad games early, and focus your efforts on the winners.

If I can make a game that does the three things listed above, I'm willing to give up all the game design traditions that don't work for design-testable games.

The result is a refactored, innovation friendly game design methodology. You can take risks and succeed. You can spend less money and still beat the big guy. As the next generation titles come upon us, the smaller game developers have a choice. They can either work smarter or they can die. Design testing is a great tool for avoiding the later.

-Danc.

Read more!