Data, distance and biases

Crime scene investigation

One of the most difficult things in a crime investigation is the debriefing of witnesses. Asking witnesses who were on the crime scene to tell you what they saw can end up in getting so many different stories that even sometimes contradict each other. So police officers need to clarify, question again and again and get to some conclusions about what happened at the specific time and place.

The same goes for tango events. You can ask the participants of an event how was the event and you will possibly get a big spectrum of answers from amazing to awful! Why does this happen? Because we all have our own experiences and expectations. Some get to dance more than others… some get to enjoy the chatting more than the dancing .. some will even complain about the food. So the work of an organiser is really difficult. I don’t think it is easily possible for organizers to make everyone happy but I guess that the more they manage to get people on the positive side of the spectrum the better the event is.

Some like it hot!

Lessons from a research experiment gone wrong

But how do the organizers get feedback? Is it credible? Are they looking at it objectively? Are they sure they do receive honest feedback? Event organizers are much like researchers in this respect. Do you know how many researchers got it wrong at some point in their experiments? Do you know how many researchers fall victim to personal bias every day? How possible is it that event organisers do not fall into any kind of bias when it comes to evaluating and improving their own event? How objective can they be in their evaluation?

As a researcher, during my PhD, I developed a solution for blind users that would allow them to navigate web pages more easily by finding elements on a page such as menus, ads, headers, etc. based on their location on the page (up, down, left, right). I searched in existing research and I saw that nobody ever tried such a solution. So I thought that it would be a quite novel approach. Although my professor warned me that the solution would not work as I imagined, I dismissed his concerns and continued to develop the solution and then test it with users.

When the results came in, I couldn’t believe my eyes. I was thinking I did something wrong in the test. Maybe I should have explained the solution better to the blind users I thought… or maybe the measurements were not the right ones… or maybe there was some other factor that influenced the results. I didn’t know but I was trying to find an explanation that would confirm my initial hypothesis that my solution would be efficient. However, the more I looked into the test results, the more I realized that something else was off. And that was ME!

Yes… You guessed it right. My professor was right from the start. The solution I thought did not work because it was based on the false assumption that blind users think of web pages as two-dimensional spaces with different areas. But that’s the view of a web page for everyone else apart from blind users. For them, a web page was just a single-dimensional series of texts. There was no up down left or right on the page… there was only forward and back in reading it. So the whole design went out the window and I continued with developing a different solution that worked (thankfully).

Lessons learnt

This experience was one of the most useful insights I ever got into biases and how they can affect your thinking. First of all, I was only open to hearing positive feedback (see confirmation bias). I would dismiss my professor’s concerns just because they did not agree with my beliefs. Second, even when the results came in, I tried to disqualify and discredit them. I tried to prove that the test was wrong and not my solution. You see… I invested time and effort and it wasn’t easy to accept that the solution did not work (sunk cost fallacy). Not to mention that the whole design was based on another biased thinking called the familiarity bias where designers/creators try to differentiate themselves from others by breaking patterns that their users are familiar with ending up with worse designs than what they start from.

When you organise and evaluate your event, it is very easy to fall into the same biases. You can easily ignore negative comments thinking they are a minority or that they are coming from lower-level dancers, uncredible sources, or any other similar excuse. You have invested time and effort and it is very hard to believe that you did something wrong. That is why you will try to find excuses for any bad comment… for any negative review you get.

Good event organizers, like good product designers, are not the ones who receive only positive feedback. No. That is easy. Who would openly express a negative opinion about an event? Out of courtesy most people would just lie (courtesy bias) and say a thank you for the organization and never come back. In the end, you are left wondering what happened to all those dancers who came by and never came back to your event.

Good event organizers are the ones who ask you first what you did not like in the event. What was your problem? After all… there must have been something that bothered you… right? Good event organizers seek for the negative feedback and dismiss the positive. Negative feedback is an opportunity for improvement and growth whereas positive is just an opportunity for stagnation.

Data and distance

Do you know what convinced me that I was wrong in my solution? Data and distance. Data because they never lie and distance because it helps you see the data objectively. Seeing the numbers of the performance of my solution compared to the normal one was a big slap on my face. A wake-up call! But data alone will not reveal the truth. You need also distance. Distance from the object you developed, distance from the event you organised, to be able and read what the data tells you objectively. That is why under protocol, police officers are excluded from investigations when they are somehow involved with people in the case. If you are unbiased, detached from the object you evaluate… from the event you organize… you can read the data objectively. Once you manage to look at your creation as if it is not yours….  as if it is someone else’s… things get much clearer. It is hard. It is even painful sometimes. But the sooner you do it, the better you will feel in the future and the better the event you will organize.

Getting distance from your event is difficult. If you can’t do it yourself ask someone else to do it for you. Ask someone who is not invested in your event, or even better someone you respect and don’t have a strong connection with, to gather feedback for you. Have people you trust to ask around and gather feedback from people in informal chats. Someone who can take the role of undercover police or detective. You never know what people could tell in those chats that will never tell you in your face.

But you can also do something even better. You can get data from a distance. The easiest way would be to simply create an anonymous online survey and send it to your participants a couple of weeks after the event when everyone is back to reality, far from the tango high of an event. See what you get back and read the data as if it was for another event. Not yours. Don’t be quick to judge, react or dismiss them. Let them sink in and then decide on your next steps. As I said… it takes courage… it takes hard decisions… it can be painful. But it is better to face the ugly truth early than live forever in an illusion.

Tonight’s Goodnight Tango

Tonight’s Goodnight Tango is the sad realization of a man that he lives in lies. Lies that made him believe that he was loved. Lies that created the illusion of love for him. Much like the lies that might create the illusion to event organisers that their event is loved.

So how about you? Have you ever been to an event and asked for feedback? Were you honest? How did the organizers receive it? Did you see any impact? Would you participate in a post-event survey? Have you ever received one? How do you like the idea?

Do you have something to say on the topic?

Did you like the post? Spread the word…