It’s important to get the survey responses/scales correct when doing a survey
This Sonos Roam Bluetooth speaker survey came across my desk today, and it uses a very unique scale that I don’t think makes a ton of sense.
Surveys can be a powerful tool when building and refining products and services, but they can also yield bad data and insights if they aren’t done well. I’m going to quickly walk through some of the issues with this scale and survey and how I would have handled it differently.
Plenty of people may do something a few times a month or year but not do something 1-4 times per week, particularly an activity like listening to music/audio content outside of the home without headphones. This is the first place this scale goes sideways. A survey scale needs to allow for people to answer the survey accurately.
I personally use my Sonos Roam mostly while at home. I only use it outside of the house when I travel, and I do not travel on a weekly or even monthly basis. If I select between 1-4 times per week, that is a significant overestimate of how often I use it outside of the home, but I can’t select that I never use it outside of the home because that’s just not true.
Sonos also asked about how often you use the Roam at home, so there was no assumption (and there shouldn’t be) that the Roam is mostly used on the go. The Roam may be a Bluetooth speaker but it is also a premium speaker that retails for $179, works on wifi for better audio quality and reliability than Bluetooth, has built-in voice assistant support, and connects to other Sonos speakers for multi-room audio.
The survey also didn’t ask about how you use your Roam outside of the home, which I think would illuminate a lot about how and why people use their Roams. And I think it would yield better data than just trying to parse how often people are using their Roams at home and away.
Understanding the why of why users do something is the most critical thing. If you knew that I use my Roam outside of the home only when traveling or on vacation, you’d know that my usage pattern outside the home wasn’t very consistent.
I also think 7-10 times per week and more than 10 times per week is splitting hairs. This scale and some of the other scales that other questions had was really splitting hairs, and I don’t think would yield very good or actionable data. And if you can’t get actionable data out of your survey, why do it?
Beyond the issues with the scale itself, the survey responses overlap. If you listen to your Sonos Roam four times per week, do you select “between 4-7 times a week” or “between 1-4 times per week?”
It’s important to have people not involved with your survey test it out first to make sure there aren’t obvious mistakes like this.
The other thing I noticed was that the survey response scales varied wildly by question. When doing a survey, you want to use as similar as scale and response type between questions as possible. I’m not going to post the entire survey, but the response scales were so varied that you’d have issues with methodical validity.
For this particular question, I’d like to see a scale more in line with this:
Multiple times per day
Daily
A few times per week
A few times per month
A few times per year
Never
There is no overlap in survey responses, and these responses allow for normal usage patterns to be accurately captured. Ironically, the scale in the survey will both lead to less accurate responses while also giving a false sense of precision. A lot of people not well versed in survey methodology, and I wouldn’t be surprised if this survey were created by a product manager or business exec with little research experience, think that numerical responses are better.
More accurate responses are better. And my scale can be reported out numerically. i.e. 40 percent of Sonos Roam users report using their Roams outside of the house a few times per week.
You could further tweak this scale, but overall it’s much clearer, and people taking the survey will be able to provide more accurate responses quicker. It’s also important to keep in mind that you don’t want people to be confused or have to do a lot of thinking when taking a survey because both of those are great ways for people to stop taking the survey.