21 Ways to Test your Survey Questions

21 Ways to Test your Survey Questions

Testing your survey questionnaire is absolutely essential for ensuring a high quality survey. There has been a proliferation of question testing methods (both new methods and variations of existing methods). This course covers 21 different question testing methods. It combines information from the survey methodological literature with practical advice and hands on practice of some of the methods.

Learning objectives

By the end of the course, participants will:

• Be familiar with the 21 methods to test survey questions
• Understand the pro and cons of the different methods
• Have ideas for multi-method and simple testing schemes

Topics

This course includes:

• Methods for testing interview questionnaires – standard pilot, interviewer rating form and variations, and behaviour coding (classical, sequence-based and shortened),
• Methods for testing web, postal or interview questionnaires – expert review, systematic form appraisal (including two online programmes), respondent debriefing (including vignettes and web probing), cognitive interviewing (very short introduction because the SRA offers courses in cognitive interviewing), focus groups, split ballot tests, and usability testing,
• A session on latent class analysis which can be used on larger pilot samples to look for error rates. (Other analysis techniques to diagnose problem survey items are described in the course material as an appendix), and
• Discussion of new and unusual techniques which can be applied when collecting pilot data (e.g., eye tracking) or alternatives to collecting pilot data (e.g., computational linguistics, crowd sourcing and soft launch), as well as traditional procedures such as examination of item nonresponse, and response distributions of pilot data. (How to measure reliability and validity, use record check studies and paradata on pilot data are described in the course material as an appendix.)

Who will benefit?

The course intended for participants with knowledge of questionnaire design and survey research in general.

Learning outcomes

Participants will:

• Recognise the importance of testing questionnaires,
• Be familiar with the 21 methods to test survey questions and their pros and cons,
• Have the tools to critique new or existing surveys and
• Have the knowledge to make better survey questionnaires that will facilitate better quality research.

Course tutor

Dr Pamela Campanelli is a Survey Methods Consultant and runs The Survey Coach business. She is also a Chartered Statistician, Chartered Scientist and Fellow of the Academy of Social Sciences. She has worked at the University of Michigan, the U.S. Bureau of the Census, ISER at the University of Essex, and NatCen Social Research. She has both led and been a team member on ESRC grants (one on survey nonresponse and one on measurement error in mixed mode surveys). She regularly teaches short courses in the UK and abroad for government departments, survey research companies, universities, as well as for various other institutions and businesses.

www.the-sra.org.uk/training