Survey Development Guide

Planning

• Define Objectives: Clearly outline the purpose and goals of the survey. Determine what information you aim to gather and how it will be used, analyzed or reported.  Do not ask for information that you cannot use, is not material to the goals of your project, or is available elsewhere.
• Identify Target Audience: Define the specific group(s) of individuals who will participate in the survey. Consider demographics, roles, and affiliations relevant to the survey objectives.
• Choose Survey Methodology: Decide on the most suitable survey method, such as online forms, paper-based surveys, or in-person interviews. Consider whether the survey needs to be confidential (the identity of the respondent is known to the survey provider, but will be concealed in reporting of results or sharing of data) or anonymous (the identity of the respondent is unknown to the survey provider), as this will impact how you are able to collect data.
• Consider Census or Sample Survey Methods: Census surveys are those that reach all members of a population of interest.  The benefit of census surveys include no sampling error and high response rates lead to results that are more likely to be considered accurate and representative.  However, census surveys can be expensive, resource intensive, and may not be feasible given the size of the group in question.  Sample surveys involve a subset of the population of interest, often making them more cost-effective and efficient.  The potential for sample error can exist if the population surveyed does not have characteristics that mirror the whole population in question.  Statistical techniques may be necessary to create a sample and/or to generalize findings from the sample to the larger population.

Development

• • • Design Survey Questions: Craft clear, concise, and unbiased questions that align with the survey objectives. Avoid double barreled questions (including “and” or “or”) which might have different implications for the respondent (i.e. Did you attend class and work?).
    • Pilot Test the Survey: Conduct a small-scale trial of the survey to identify any issues with clarity, relevance, or functionality. Adjust questions or survey flow based on feedback before launching it to your full population.
    • Select Distribution Channels: Determine the best channels to distribute the survey to the target audience. This could include email, social media, university websites, or physical distribution points.  Consider whether your survey is anonymous or confidential when selecting methods of communication because this may impact how or where you advertise.
    • Set a Timeline: Establish clear start and end dates for the survey to generate a sense of urgency among participants. Allow sufficient time for data collection while avoiding survey fatigue.

Survey Question Format

When developing a survey, the format of the question is as important as the content. Consider the following pros and cons of these popular question formats as you develop your instrument:

Multiple choice or closed-ended questions

Pros:

• • • Efficient and quick to answer.
    • Responses are easily quantified, analyzed, and presented.
    • Provides easier comparability across respondents.

 Cons:

• • • Limits respondent to predefined options
    • Options may not be exhaustive, including the full range of potential responses
    • Poor design or wording can lead to misinterpretation.

Open-ended questions

Pros:

• • • Provide opportunity for rich qualitative responses
    • Can uncover unknown or new perspectives and information
    • Provides opportunity to explore complex issues

Cons:

• • • Analyzing qualitative data can be time-consuming or expensive
    • Analysis can be subjective and susceptible to bias
    • Standardization of responses can be difficult

Matrix questions
Pros:

• • • Allows for group of similar question content and/or format
    • Space efficient
    • Easy to understand, thereby reducing burden on respondent

Cons:

• • • Can be overwhelming to respondent, depending on number of questions
    • Order of questions/content can influence responses
    • Respondent may check same response in repetition
    • Output from survey tools may be difficult to analyze.

Likert Scale questions
Pros:

• • • Provides consistent scale of measurement
    • Responses are easily analyzed and quantified
    • Common format is easy for respondents to use

Cons:

• • • Scales can be difficult to interpret
    • Providing a neutral response can end up as a default for respondents
    • Format does not fully capture nuanced points of view

Semantic Differential (Attitudinal) Scale questions

Pros:

• • • Can measure attitudes based on contrasting adjectives (“Supportive” vs. “Unsupportive”)
    • Polarity of scale is easily understood
    • Data is easy to present

Cons:

• • • Can only be used for simple or binary dimensions
    • Cannot be used for complex concepts
    • Interpretation of the scale may differ between respondents.

Execution

• • Encourage Participation: Develop strategies to boost survey participation, such as incentives, reminders, or messaging from university leaders or recognizable figures.
  • Incentives: Depending upon the population and needs of your survey, it may be desirable to offer incentives to survey respondents. Two guidelines to consider are that the incentives must be positive (i.e. no negative consequences for non-respondents) and should be relevant to the topic of the survey (e.g., if you are
    conducting a transportation survey with the purpose of encouraging public transit, free parking passes would not be an appropriate incentive).  If planning to use gift cards, please refer to BU Gift Card Policy to understand the tax implications and purchasing requirements.
  • Data Collection: Monitor survey responses regularly to track participation rates and identify any technical issues. Ensure data integrity by implementing measures to prevent duplicate or fraudulent responses.
  • Firewalls and Whitelisting: If conducting an email survey from a source other a BU email address, it is important to contact Information Services & Technology well in advance of deployment to make sure that those emails will not be filtered by university firewalls preventing spam email.

Analysis

• Data Analysis: After you have collected the responses to your survey, it will be necessary to analyze the results. Simple analyses can be done using Excel with Pivot Tables, and more advanced statistical analyses will require the use of a statistical software package. Site licenses for SPSS, SAS, and JMP are available for download or purchase from BU Information Services and Technology.
• Transparency: Once your results have been analyzed, it’s important to share the results in a way that they are accessible to the community surveyed. Being transparent with results allows the populations surveyed to see the value in answering surveys.
• Interpretation and Reporting: Interpret the findings in the context of the survey objectives and university goals. Prepare a comprehensive report summarizing key findings, recommendations, and action plans.
• Feedback and Follow-up: Share survey results with relevant stakeholders, such as university administrators, faculty, staff, and students. Seek feedback on the findings and collaborate on implementing changes based on the survey insights.
• Continuous Improvement: Use survey feedback to continuously improve future survey efforts and enhance the university’s overall operations, services, and student experience. Regularly revisit survey objectives to ensure alignment with evolving priorities.