Gå till innehåll

I just finshed reading Rocket Surgery Made Easy. As the title so clearly says it is a do-it-yourself guide to finding and fixing usability problems.


The author´s name is Steve Krug and here´s what he looks like according to the book. skrug_cartoonSo next time you happen to be at a usability conference make sure you say hello...

My current personal goal is to become really good at usability testing so I try to read good books and browse the web for videos, talks and other useful stuff like templates for creating prototypes. This post will focus on my interpretation of what Steve says.



Have You Ever Met a Good Midget Comedian?

This book is just like a really good midget comedian... very short and very funny. It is full of headlines like this one together with those nerdy quotes that we all love to hate and cherish so much. The title may as well have been Usability testing made entertaining. For a long time I tried to make usability a part of the IT-projects I worked in with limited success until I decided to keep it very simple and do it myself. If there is something I have learned from 20 years in business it is that large, complicate and cumbersome stuff never ever gets done! At least not for real or for very long.

In my opinion, the essence of the book is:

      "Let´s make it so easy and still so rewarding that it´s very hard to motivate not doing it!"

Even better, the book in itself is made accessible so there is hope that even the most reluctant readers can get through it. To be really clear, this is a book on how to get you STARTED, not the comprehensive works that can be found elsewhere. The problem with most thicker books is that usually at least half of the words are totally unnecessary making the reading experience not very pleasant and then there is so much information that it is really hard to find out what is most important. So cudos for effectively distilling usability into something more devourable.

So by now you have realised that I really liked this book! I like it so much that I will most likely use it as course litterature for a one day usablity class I am planning to create. I would love to translate it to Swedish to make it even more accessible fo my fellow countrymen.


Rocket Surgery Cover Page
Rocket Surgery Cover Page


The Core Process of Usability Testing

Let us start with the goal of usability testing made easy:

Our goal is to find and fix the most serious problems in order to make products easier to use.

That´s really all there is to it. We don´t try to find all of the problems and we do not bother about statistical significance in our measurements. The point is that bad usablity shows itself readily and even if we found momore problems, there would be no time to fix them. So if we can find what is hurting the most and fix this right away, we stand a great chance to end up with a great product. This type of usability testing has one large advantage over more advanced versions - this one get´s done(hopefully)!

If you look at the tis on planning and execution it is pretty clear that this is usability the agile way - and agile is good for us, right?

When Do We Start

"Start earlier than you think makes sense" (S Krug)

Right along the lines with Lean UX we start testing early and continue throughout the project. We have a hypothesis and we test it to see how well it works. How? Create a prototype and do a usability test. It is soooo easy to create a prototype in powerpoint or Omnigraffle that it is criminal not doing it! It saves so much time if we test before the developers have started coding. Of course we need to do some background work beforehand mapping the user experience or creating an impact map so we understand what the real problem is. This is not discussed in detail in this book but an earlier blog post. is a good start for learning more.


One Hour Well Spent

A test session takes only one hour, an extra half hour extension is possible but in my experience that may be too long both for booking people and for keeping focus.. Make sure that everything is well prepared. Since we have only one hour to spend, there is no time for last minute fixes like not having your test data or equipment ready. The easiest possible configuration is to have a device with your application loaded, and one test manager telling the participant what to do and then observing and taking notes. The recommendation is to have at least one observer - in the same room if you can keep them from talking during the test. I find it a bit hard to take great notes at the same time as I am leading the test.

Now here is an important point that Steve makes. Having more observers gives us many advantages:

  • people make different observations
  • it is a great learning experience for everyone involved
  • it helps getting team consensus on what to fix
  • the understanding for why usability testing is needed increases

These additional observers should be kept in another room since too many people in the same room make he subject feeling watched - which is a correct observation - and that will disturb the testing. So hook up a big screen using GoToMeeting or any other tool you have - Skype or Project Place can work although I have found Skype a bit choppy at times. Use an external microphone and extra speakers to record and replay the talk aloud and instructions. No need to record the face of the tester - audio is enough. Recors the screen if you think there is actually a chance that you will watch the re-run. It can of course be good ammunition for pointing out a problem later.

Now introduce the tester to what they will do and make sure they understand we are testing to see if OUR DESIGN HAS PROBLEMS not if they are skilled! Don´t forget this and don´t tell them more than three times in a session since this will lead them to think the opposite.

Set up of simple think aloud session
Usability testing using think aloud

Now it is great to have another test lead in the observation room to distribute the tasks in written format together wih a place where the observers can write what they see and sum it up in their top three problems.

Post Test Session Processing

Meet with all of the observers. Collect all top three observations. I like to do this on post-its. Do a collective sorting and prioritising. And voila, you have a list of most important things to fix and some other stuff that you will not fix right now. It may seem a bit provocative to some - but if you have not observed - you don´t get to vote. Cause we are not voting on what you think is bad - we make the prioritisation on what you actually observed! This will hopefully make more people wanting to observe. Time well spent I think!

So that´s it. No big report that noone will read, no huge list of to do that we have no chance getting done. Pretty agile, huh!


Final Words

Last project I worked in failed miserably at the usabilty testing part. Lack of understanding and knowledge was the biggest culprit. If was to take the advice from Steve I would have run tests from early on although the web interface looked crappy for a very long time. I would have created a quick prototype on my own to start with just to have something that enabled me to get feedback. I would also have used my colleagues and friends as testers since many usability problems have to do with general things like navigation and information that also non-business specialist will run into. I would have recorded sessions and shown them to the members of the project.

I was so pleased after reading this book that I bought and read the updated version of Don't make me think. This is a good read as well but more about that later...



Scrum does not work. I agree that thins are not working but are we really doing Scrum?

- That Scrum Stuff Does not Work!

… my highly regarded developer colleague exclaimed a while ago. My reptile response was that Scrum Does work – if you do it right. Then I thought some more and realised from my 20 years of working in IT that “People working together” is what really matters”. Regardless of what method was the latest fad and eagerly promoted by people on the front line – sometimes myself on the barricades – when I got in a project with the right people, things got done. I will not deny that a lot of the agile thoughts resonate well with my own experience. I feel an urgent need to share some stuff with you, here it is!

People working together

If we get people really working together problems are more easily solved. Whenever I have had the chance to work in a team that continuously discuss and solve problems together, we have performed well. Needless to say you need skills in order to perform, but skills combined with an arrogant know-it-all-myself attitude seldom (never) leads to success. Actually the opposite is often true. Pebbles in the machinery can be handled but if you get a big rock in the wrong place it can drag the whole team to underperform. I have seen it in different roles from developers or testers claiming to have THE only right answer, requirements analysts that are much reluctant to change THEIR wonderful document or even discuss what you find unclear and project management the old style that kills argumentation with a distinct – Now, this is the way we are going to do it!

My favourite quote is “Tobbe, if you do not do what I tell you to do - you are not a team player!” I mistakenly had though my role as test lead was to question things and to do my best and if I found something that looked really wrong I should act on that. But that turned out to be valid only as long as I did not question the project manager. As you may understand, authoritative leaders, especially those that try to tell me in detail how I should work claiming seniority since they are much older, more experienced, have a degree or certification in something and also once upon a time long ago did some sloppy work - then known as testing – have a really hard time working with me. I do not claim to know everything and I am not trying to make you look bad but I DO want the team to deliver and to do my best. If I know something and I would like to discuss this with you and if you then put me down I come back as a rubber ball on steroids.

As tester and analyst my job description includes questioning what is right. In a healthy project the team wants to perform well. Questions, if they are posed in a respectful way, are readily discussed and agreed on. If you see signs of people more wanting to protect than to discuss their creations or their leadership you are in for some serious trouble.

The same goes for vehemently arguing over the exact process rather than the best way to solve a problem. It is probably a good idea to first adopt then adapt Scrum as a process instead of trying to make a lot of changes without trying the original thing first. A lot of bright and experienced people put a lot of effort in creating Scrum so what makes you think you know better? Read about Johanna Rothman ranting about that.

Embracing change

Not too many years ago a fellow tester exclaimed “If we can just get complete and correct requirements from the beginning, our testing work would be so much easier” Then we had this long argument where I claimed that that is a totally unrealistic expectation since it is built on the following premises:

  1. All client stakeholders have managed to internally understand and  agree from start exactly what they want
  2. The customer or a requirement analyst are able to formulate this complete and correct view in a way that leaves no room for interpretation
  3. It is technically possible to fulfill the requirements
  4. During the project the customer never wants to make any changes based on new needs, cost estimation of what we are supposed to build or any other inpput

Well since many of us have realised that all of these premises are unrealistic and are regularly violated in each and every project we work in we prefer to leave the requirements open for discussion.

Certainly there have been projects where the requirements were more known from the start of the project than others. We did build things before agile in a more waterfall like manner with a one document requirement spec and did succeed pretty well in some cases. A recent project I was in tried to do it as a mix of waterfall and agile – all of the requirements were written down as a start but since we were going to work in Scrum the readable document was split up in user stories. Then we all conveniently forgot/ignored the very important point regarding a user story – that the team has to have a conversation with the customer about what it really means and agree on acceptance criteria. Instead the requirement analyst wrote the detailed requirements in acceptance criteria format from the start. That was a great idea since he then could move to another project and only in rare cases be bothered with any clarification work. Needless to say, we all hated the requirements and the developers did not even bother to read them! It is a far too common misunderstanding that we are using a method just because we are using a template connected to that method. A good place to start is this article regarding the different approaches to requirements. http://www.scrumalliance.org/community/articles/2010/april/new-to-user-stories.

Working Closely with Your Customer

Another important guideline for system development is that we have close collaboration with the customer throughout the whole project. Scrum recommends small collocated teams because we need to discuss in order to understand each other. I do not think that this is something that came with agile rather that has been common knowledge for a long time. Unfortunately common knowledge seems to be rare at times.

In my experience actually sitting together with the customer or user is often impossible due to their constant lack of available time. That does not mean that we cannot still work together on a weekly basis. The most successful projects I have been on have all had very engaged customers and requirements analysts. We have met the customer every week to clarify problems, if the customer has not been able to visit us – we have sent somebody to meet in a place at their convenience. The most successful meetings have been where the user/customer meets the whole team. That is usually at the demo and the sprint planning meeting.

Task flow, prototypes and Teamwork

A problem we have had with demos is that they sometimes become like a scripted test were a few selected flows are shown and all open questions and remaining bugs are avoided. To remedy this we have successfully done two things.

  • Acceptance test environment where customer is encouraged to play around on their own
  • Run usability tests after each new delivery

These two fairly simple things we have found give a lot more feedback than the demo and are good value for the time spent.

To any product owners or users that decide they do not have time to participate in the process I say that is a really bad idea and will cost you a LOT more in the long run. You complain that IT drives the project and engineers that do not understand the true user do the design – well start communicating with us and help changing that!

Make Sure you Focus on the User Experience

I have heard the argument that when releasing often usability is a non-issue and I must disagree strongly. There also seem to be a misunderstanding that Anyone can do UX and I disagree again. Just because something does not require in depth technical knowledge doesn’t automatically make it easy. I have heard the same argument about testers, requirement analysts and project managers – if it does not require an advanced tool anyone can do it Well there is a big difference between doing something and performing something really well. Yes I do have a camera and some photos turn out really well but most of them are not appreciated outside the immediate family. Finding out what a user really needs is a laborious task that requires active observation, good interviewing technique as well as an analytical mind and the ability to be a link between users, business and IT.

There is a very real need for UX and development to agree on a suitable level of what should be ready before coding starts. The agile idea is to avoid BDUF, Big Design Up Front and the UX idea is to first finding out what users real problems are before starting to solve them. This includes some early prototyping which is done in the discovery phase of Scrum. UX then need to be a part of the team building the solution since everything will not be set from the first day of coding. A UX designer can be very helpful in answering questions regarding proposed solutions and someone will have to lead the usability testing effort.


In a recent project the choice was to completely ignore any usability questions and to instead focus on functionality. I had to argue for making fields large enough to show all the data, you have to think I am joking, right? The whole idea of separating functionality – like selecting the right field in the database – from usability - actually being able to see the data and understand what it means – must be flawed!

Most developers I have worked with are neither skilled nor very interested in usability or UX questions. If you are a developer and this does not apply to you, great! For the rest, at least recognise that the team needs to have the competence of building great wireframes, interaction design and usability testing. Things that have made our applications look and feel professional and usable are:

  1. Making sure that the business analysis is done properly. Experience mapping used in UX works great. This effort in the discovery phase is the back bone for doing the right thing.
  2. Professional help with wireframes from someone that can connect the UX with the solution, not only looking good
  3. A specialist building at least the first version of the CSS
  4. Running usability tests on a regular basis

I recommend reading the principles behind agile - not only the manifesto. It starts with:

Agile principles
First agile principles

User Experience and a good collaborative team is a great start if we want to honor the principle above.

So does Scrum Work or Not?

Just because you have a Scrum board does not mean you are doing Scrum! My advice would be that you first try doing it the proper way - then you can adapt it. To my developer colleague I simply stated " You cannot say Scrum does not work because we are not doing Scrum"  

It is a simple yet powerful framework that I like to have as a tool in my own toolbox. Combine it with other good practices like Experience Mapping, Continuous Integration and Exploratory Testing.