Design Technique 4 – Making sense of observations

Once we make the game for the device and know that it is functioning as we want it to, we need to find out if the user also reacts the way we want them to. To do this, we hand the device over to them loaded with the game and observe while taking notes.

There are two different ways in which we can make observations, obtrusive observation and unobtrusive observation. Unobtrusive observation means you simply observe what the test user does and refrain from interacting with them, with this method you realise whether the system is user friendly and if it is ready to be published. In no way can you influence what they do by explaining designs or asking any questions.

Using the obtrusive observation method means you are able to interact with the test user, e.g. by asking questions. Using the obtrusive observation method, you are able to learn more about the usefulness and acceptance of the system.

By carrying out these tests, you can evaluate the ease of use and learning for the game. During usability testing, a real future user should be used to give you the most accurate results. I don’t think neither method is wrong, but to get best results you should use a combination of both tests in an efficient manner. For example, each time you start a usability test you could start with the unobtrusive observation: you observe how users execute the tasks you give them. After that, you reserve sometime to ask some questions, explain the design process, and answer the test user’s questions.

I’ll be going into a little bit more detail on both of the methods so that the differences are a bit clearer.

Unobtrusive Observation:

1. Observe: be quiet, watch and understand

- Don’t explain

- Don’t ask the test users opinion

- Don’t defend the design

- Don’t apologize

- Don’t suggest

- Don’t contradict the test user, nor agree: stay neutral

This is actually harder than it initially seems. It is in our nature to help others when we see them in trouble, even more so when it is with a design we have worked on ourselves. Just keep in mind that you are there to learn about the test user’s behaviour and not teach or convince them.

2. Only help to overcome the limitations of the prototype

Explain briefly and in a neutral way what would happen in the future system.

3. First observe, then take notes

Don’t let your note taking get in the way of your observation; you'll end up missing some details. You don’t have to write down every observation you make, just the key ones. As soon as the observation in complete, take a minute to write down anything you may have not already while it is still fresh in your head, wait too long and you’ll forget the important details.

4. Stimulate the users to think out loud

But use neutral points, e.g. “What do you see, what are you thinking, what are you looking for?”

5. Limit the time test users have to execute a task

Don’t prolong the test users suffering more than necessary. If the test user is stuck on a certain task and you have learned of the reason, get them to continue.

6. Extract detailed information

Test user: “I see a lot of information.”

Observer: “Could you tell me what information you see?”

7. Answer test questions with questions

Dealing with test questions is probably the most difficult part of unobtrusive observation. Before the test you should explain to the test user how you will be conducting the test and how their questions will be dealt with. You should encourage the test user to ask you questions as they will help you figure out what is not clear with your system, and also that you will only write them down for now and answer them at the end of the observation.

Test user: “What does this text/symbol mean?”

Observer: “What do you think it means?”

Test user: “Should I click here?”

Observer: “What do you think happens if you click there?”

Obtrusive Observation:

1. Think of what you want to ask them before the test

When you have just completed you unobtrusive observation with the user, you will obviously have questions from them which you will need to answer. But you must also have your own checklist with things which you would like to know from the user, e.g. functionality, acceptance, etc.

2. Ask open questions

Try and avoid closed yes/no questions, by asking open questions you will bej able to draw out much more detailed answers.

Bad: “Do you understand what this means?”

Good: “What do you think when you see this?”

3. Don’t blame the user

Remember, your testing the usability of the system, not the computer literacy of the user. If there is something the test user does not understand, then it is something that will need to be reworked in the design.

Bad: “Why don’t you understand this?”

Good: “Could you tell me what this mean for you?”

4. Don’t ask the test user for design solutions

Test users are not interaction designers, that is why they will not provide you with any good solutions. Don’t bother asking, but take notes of suggestions they make spontaneously.

Bad: “Do you need a button here?”

Good: “What information would you require at this point?”

5. You can do obtrusive observation with groups of test users

You will get much more feedback using this route, but it may end up being a little overwhelming. People often find it easier to form their opinions when they are confronted with the opinions of others. You can also learn much more on the usefulness of your design as well as its acceptance amongst the group as they discuss it. Your role will be to facilitate and focus on the discussion.