Stars, Hearts, Smileys, Buttons or Grids: How to Design Rating Scales

Rating scales are one of the most frequently used type of questions in surveys. There are many ways to present the items of the scale. Many times, traditional radio buttons are used in grid or matrix formats. With more and more people accessing surveys via mobile phones, online surveys are being adapted to fit small screen sizes. In addition, rating formats from outside survey research, such as the use of hearts, stars, and smileys can be used.

In an experiment, we randomly distributed respondents over a five-point Likert scale, varying the response format. We use traditional radio buttons, fully clickable tiles, grids, and visual response options such as smileys, stars, and hearts.

We discuss responses quality indications such as response rates, frequency distributions, satisficing, primacy effects, and respondents' evaluation of the survey. In addition, we compare surveys across devices: regular desktop PC, tablet, and mobile phone. We focus on special populations in our discussion on which formats works best for which population.

Data come from the panel of Internetspiegel. This panel consists of civil servants in the Netherlands. Preliminary results show that smileys work best.