The site requires Cascading Style Sheets (CSS) be enabled in your web browser

Effective Survey Design Notes

Introduction

This guide contains information for designing a successful survey using the DCA Survey's survey construction tool. It provides a good introduction for both beginners and experts designing a survey for the first time using this website.

Goal Setting and Defining Objectives

Before you actually start designing your survey, you should ask yourself the following questions:

Reflecting on and answering these types of questions will help define your goal. Most likely you already have a goal set before deciding to conduct a survey.  Once you have a goal in place, you will then need to define one or more measurable objectives. These objectives will be used to determine if the goal has been reached.  Using a survey is an excellent way to tangibly measure those objectives. With DCA Survey's online surveys, you receive survey response results immediately, and in real-time.   This is beneficial to you as you will have current information at your disposal to measure your objectives.

Data, Information, Knowledge

Now that you have objectives set, what you now seek is information. You will use a survey to collect raw data from survey respondents. The data collected, in aggregate, should provide you with information. If the resulting information is relevant and accurate, your surveying efforts will be a success. It will provide you with information of a higher order: knowledge.

It may take a follow-up survey or two to ultimately achieve your goal. You will know best when reviewing the original or perhaps each iteration of survey results.

Planning and Preparation

The key to conducting a successful survey starts with planning and preparation work. You will need to think about what survey questions should be asked, how many, whom you will target as survey respondents and what the survey's presentation format will be.

If the survey subject matter is not familiar to you, it is recommended you first do some research. Try to learn as much about the topic of interest. It will help you in constructing your survey questions and you will know better who is your survey's target audience.  A credible survey makes for a better response rate.

A more detailed discussion concerning the creation of survey question is included in this document. One important and immediate consideration to note is that if the questions ask respondents things they do not know or understand, then it may result in inaccurate data. If so, flawed information will result. It is important that you target the proper survey respondent audience. It is highly recommended you perform a test survey, first, to a smaller audience (sample).  The feedback provided from this important step will confirm that you will be targeting the proper audience (population).  Using the test results from a small audience, of about 5 to 10 people, you will then revise accordingly before performing a full fledge survey study

You will also want to give some thought as to how many questions to ask and how they will be presented in the survey.  With a plan ready, you will use the survey construction tool to apply what you have in mind. The DCA Survey survey tool contains many different question template types as well as other robust features.  Such features include whether to group questions into pages, require a response or not and whether to randomize questions or answer selections, to name a few.

State a Purpose

You will be soliciting people take your survey. This requires something everyone values: their personal time. If the purpose of the survey and how it is of benefit to both you and them, is not obvious, please be sure to communicate that important information to your target audience.

Telling people the reason why you are doing something is one of the most powerful influencers of human behavior. Robert Cialdini, Ph.D. in his book "Influence: The Psychology of Persuasion" writes about an experiment performed by Harvard social psychologist, Ellen Langer, which concludes people like to have a reason for what they do. Communicating the purpose of the survey is of fundamental importance.

The "reason why I am conducting this survey.." should be the first piece of information communicated. And yes, this is true even before the first survey question is read. If you give your target survey audience a "reason why" explanation, they will be more inclined to act - as in complete a survey response.

Question Design - General Recommendations

Survey questions should be written that are clear, concise and direct. By doing so it will help the survey responder know exactly what you are asking. You should review each question's wording style and tense for consistency. Avoid colloquial phrases, slang and if possible, technical language. Omit needless words while favoring usage of simple words and expressions.

Your focus should be on either the least educated or most typical respondent while creating your survey questions. They should have no difficulty understanding the question being asked. Well understood questions are more likely to be answered and answered correctly.  This will result in accurate data collected.

Close-Ended Questions A close-ended question presents the survey responder with a fixed set of selections to choose a response. The fixed set may contain a short or long list of potential selections. Close-ended questions are favorable to survey responders as they provide a quick means of providing a response. They are an excellent choice to consider when improving your response rate. Close-ended question are also easy to tabulate, summarize, and visualize the results.

Open-Ended Questions An open-ended question provides the survey responder to provide a free-form response. This is in sharp contrast to a close-ended question whereby the survey responder must choose from a fixed selection-set of choices. Typically a user will type either a short response in a "text box" or may provide a longer response if the survey question provides a "comment box". Typically an open-ended question will solicit a response that asks for the responder's feeling, like or dislike, attitude or opinion. 

Some drawbacks to using an open ended question are that it requires more time and effort to complete a typed response than one which requires a selection. The responder may also be hesitant or have difficulty expressing their thought, and instead choose to skip the question. Another issue is that the open ended question's results require more time and effort to review and interpret results.  Open-ended question results lend themselves more commonly to qualitative than quantitative results. In contrast, the results from close-ended questions are more easier to enumerate, summarize, and find common patterns among other questions' results.

Excellent Survey Question Qualities

An excellent survey question exhibits the following qualities:

Simple, Brief, & Direct: Shorter questions take less time to read and are more likely to be answered. They are also less likely to be misinterpreted. Omit needless words, abbreviations the use of technical jargon while favoring words that are familiar to your survey's target audience. Your questions should use a consistent tone and style which will increase the familiarity and comfort level of the survey responder as they progress through the survey.

Specific: Your question should be focused on asking for one specific item of information. For instance, a bad survey question asks, "Do you like the price and quality of the Acme RoadRunner 4 door sedan car?" If the response is, "No", then the survey owner will not know if the responder dislikes the price or quality or both. In this example, it is best to break the question into two to inquire about the price and quality, respectively.

Objectivity: In your desire to seek new information, you strive for objectivity in your survey. You do so by choosing your words carefully. Care must be taken to not lead responders into providing an answer you would like to receive. A classic example of a leading question is one starting with a negative verb like, "Don't you think all men and women are created equal?". The question should be assumption free and should also avoid loaded and emotive language.

In addition to the question's text, the set of response choices provided for the question should also convey objectivity. The set of answer selections should not convey additional information for the question. Each choice provided in the set of answer selections should also have equal weight.  Likewise the location where the selection appears in the list of choices should not be influential as well, e.g. primacy effect. It is worth noting the DCA Survey tool provides an optional "randomized answers" configuration for your consideration which addresses primacy effect concerns.

Confidence: Survey responders must have a sense of confidence when participating in your survey. If a responder is concerned about the consequence of providing a particular response to a question, there is a good possibility that the response provided will not be truthful. This will have a negative consequence with the accuracy of your data collected. Anonymous questionnaires that contain no identifying information are more likely to produce honest responses than those identifying the respondent. If your survey does contain confidential items, be sure to clearly state your policy on confidentiality.  Survey questions which exhibit respect, professionalism, and confidence, will surely be reciprocated by its participants' responses.

Logical Flow: When a survey responder completes one question and then moves onto the next, the transition between questions should be smooth. This is much like writing an essay or a letter, whereby there is a logical progression from one paragraph to the next. Be sure to group questions that are similar.  It will bring greater comfort to the responder and make the survey quicker and easier to complete. In turn, you may expect a higher response rate. The DCA Survey survey design tool provides you with the functionality to group questions by pages.

All Answers Provided: Questions which contain a fixed selection of answers are most popular as they are generally the easiest for a respondent to answer. They are also the easiest results to summarize and analyze. Providing a question which does not accommodate all possible responses may confuse and frustrate the respondent. You should consider providing an "Other" response selection.  Another option to consider is providing an optional input text response box to collect a user's custom response that is not provided in the fixed set of selections provided.

Mutually Exclusive Selections: For a survey question which is designed to solicit a single selection, the selections provided must be mutually exclusive. There should be no ambiguity provided by the question's selections. Only a single appropriate choice should be provided for the responder to make. For instance, if you ask responders where they do their Christmas shopping, they may become confused when the correct response for them is more than one place and spans more than one answer selection, i.e. they shop at the mall and online. If there is potential for overlap or nonexclusively with a single select type question, it is best to provide an "All of the above" type selection. Alternatively, you may consider designing the question to use a multi-check box selection for its response.

Survey Length

As a general rule the more questions a survey asks, the fewer responses will be received. Try to keep your survey brief. In fact, the shorter the better. Response rate is the single most important indicator of how much confidence you may place in the results. A low response rate can be devastating to a study. Therefore, you must do everything possible to maximize the response rate. One of the most effective methods of maximizing response is to shorten the survey.

If your survey is over a few pages, try to eliminate questions. Many people have difficulty knowing which questions could be eliminated. For the elimination round, read each question and ask, "How am I going to use this information?" If the information will be used in a decision-making process, then keep the question as it is important.

You may also decide which questions to eliminate when conducting a test survey with a limited audience. Based on the results and feedback provided, you will have a good idea which questions are most invaluable to include in the full-blown, live survey. The DCA Survey's "copy survey" feature makes the task of copying and modifying a test survey for a real, live survey use, easy.

If you really need to ask more questions, keep the questions text and the answer selections brief. Also, it is recommended you use the "survey completion status" design option provided by the DCA Survey survey design tool. Having survey participants notified of their progress while providing answers to questions, they will be more likely to complete the survey knowing how much remaining effort is left to do so.

Ask for personal information, only if needed.

Ask individuals to provide you with personal or demographic information such as age, race, income level, education level, etc., may turn-off some respondents to the point of them deciding not to complete your survey. However, in many instances, this information is necessary for you to achieve your objective. If you must ask for this type of information it is best to place this type of question at the end of your survey. The DCA Survey survey tool provides a 'demographic details' question type selection that will satisfy such survey question design and data collection need.

Survey Layout

The survey's layout and design are very important to its success. A survey that is organized and formatted well will positively influence your survey's response rate and the accuracy of the data collected.

Purpose: It is recommended that you state the purpose of the survey to its responders. This may be done within the email solicitation, the link to the survey form, or in the survey form's introductory text. In short, the survey's purpose should be introduced before the first survey question is presented to the responder. Other information you should consider including in the survey introduction are: the organization conducting the survey, confidentiality and privacy information, and how the data collected will be used. (Note: for United Kingdom users, this is necessary to comply with the Data Protection Act 1998) By providing such information when introducing the survey, it will put responders at ease to take your survey. They will also be more inclined to provide honest answers. Honest answers, in turn, will reflect accurate data.

Expectation: In the survey's introductory message, you may also consider informing the survey participants about how long it may take to complete a survey response. Likewise, also consider using the "completion status bar" survey design option. It provides instant feedback to the survey responder to indicate how much of the survey has been completed and how much remains. The amount of time to complete at the start and during the survey will provide greater assurance to the survey responder. She will be more inclined to complete responses to question and do so without haste.

Incentive: If you are offering a prize or some other incentive, be sure to indicate such in the survey's introduction. If you provide this information in the beginning of the survey, then more completed surveys, and ones containing honest responses, are likely to occur.

Do not offer anything you cannot deliver. If it is a younger audience, such as students, consider an incentive like an electronic game or iPod prize. For an older audience, employees, and customers, a monetary gift or coupon may serve as a better incentive.

Navigation: Navigation within your survey, whether it be from one survey question or page to the next, it should be intuitively easy for your survey responders. For most participants, this will already be the case and the DCA Survey survey design tool addresses this design detail seamlessly for you. For inexperienced web surfers, you may want to consider providing introductory text of how to navigate within the survey. The DCA Survey survey design tool provides you with the option of setting the button face text for the Next & Previous (page/question) and the Submit (survey) buttons.

The DCA Survey survey design tool provides many customization settings to make your survey design look visually appealing. Such settings include changing the font size, text and background colors, and numbering of pages or questions or both, to name a few. When you click on the 'Preview' menu link, it will provide you with instant feedback on how the constructed survey looks and behaves. You may click the 'Preview' menu link anytime during the construction of your survey.

Below are some general guidelines for creating a survey with a top-notch presentation:

Answer Choices Ordering

The DCA Survey survey tool provides the option to display answer choices in a random order each time a  respondent requests the survey form. You may want to consider this survey question design option to minimize the "Primacy Effect".

The Primacy Effect is the tendency for the first items presented in a series to be remembered better or more easily, or for them to be more influential than those presented later in the series. For instance, if you listen to a long list of words, it is more likely that you will remember the words you heard first than words that occurred in the middle.

Test the Survey

Once you have created your survey, and are now ready to conduct your study, you should first gather some feedback from a small group of people (5 to 10 should be suffice). This pre-release step will ensure that respondents clearly understand the questions you are asking.  In turn, this consideration will ensure that you are accurately capturing the information that you need for your study. It will also provide you with an estimate of how long it will take an average responder to complete the survey. You will want to mention this piece of information in your live survey to help set expectations for survey responders.

DCA Survey's "Preview Survey" feature provides you the ability to simulate taking a survey without actually recording their responses. You should revise your to-be-released survey accordingly based upon the feedback provided by the test survey. After gathering the feedback from your test survey, it is recommended you use the DCA Survey's copy survey feature. It copies the survey content and structure, but not its responses. In other words, when you copy the test survey, you will be able to make clean revisions to the currently response free, soon-to-be-released survey.

You should consider the following questions when conducting your test survey:

Anonymity

An anonymous survey is one whereby nobody, including the survey owner, may identify the respective survey responses. It may be preferable to conduct an anonymous survey in consideration of maximizing your response rate and improving the accuracy of the information collected. Folks may be more inclined to respond and give an honest, accurate response if they know their responses are not being personally tracked.

You may create an anonymous survey using the DCA Survey survey construction tool. There is one caveat. By default, I.P. (Internet Protocol) addresses of the survey responders are not displayed along side of their respective survey responses. You may change the default option to display I.P. Addresses in the survey results. The I.P. Address is voluntarily provided with every web page request by a user's web browser, including the submission of a survey response.

The I.P. Address has the potential to identify a user, but only if you actively chose to do so. One example is you perform a cross reference lookup of a survey responder's I.P. Address with another non-anonymous conducted survey, to find matching I.P. Addresses. In this scenario, there is no guarantee the responses came from the same person. For instance, multiple responses came from a kiosk or a shared computer and an I.P. Address is not unique to an individual. In simplest terms and perhaps it is an oversimplification, you may think of an I.P. Address as being akin to a telephone number for the internet.

Pay attention to non-responders

When people do not respond to a survey, this passive feedback is of great potential value. It may be worthwhile to follow up and ask the non-responders why they were reluctant to answer, if possible. The survey responses people do not give may offer a valuable clue to the answers you really need to know.

Use Incentives

Consider providing an incentive to your targeted audience such as a store discount, a free training class or a free movie pass. Everyone likes incentives.  Like giving a "reason why", an incentive provides more motivation for completing a survey response.

Close Survey On-Date Usage

DCA Survey provides an optional survey cut-off date setting. Knowing your target survey responder audience best, you should use your discretion in implementing this setting. The down side is it may reduce your survey's response rate. That is, some folks may miss the deadline. The upside is that it may encourage procrastinators to actually complete the survey knowing that there is a firm deadline.

If you forego using the Close Survey On-Date setting, you will want to eventually close your survey eventually, and do so without having to announce it publicly. When you complete your study, you will want to lock-down the survey, meaning to prevent future responses from trickling in, so that the final survey results will remain unaffected.  The DCA Survey tool makes this survey setting task simple to complete. 

Analyzing data collected

After the survey has been completed, you will then want to look at the results. When this happens you will want to revisit your originally stated objectives. More simply, you will want to think about how to make use of the information inferred from the data collected by your survey. The DCA Survey naturally makes results available to you.  Additionally, if you permit, these results may be made public and shared with other people. When you have finalized your findings, consider publishing them on your website or somewhere where your respondents will be able to see it. It is important that people feel that the time spent giving you their views was worthwhile.

Response Rate

The survey response rate is simply the percentage of people who provide a survey response.

  Completed Survey Response Count / Solicitation Count = Survey Response Rate

The more responses received, the higher the response rate. If the survey is well designed and targeted well, this will also result in more accurate data as the survey results (sample) will be representative of the population.

Further discussion of this topic, including tips on improving the response rate, may be found in our Survey Response Rate document.

Target Audience & Sampling

Inviting the right audience to participate will not only positively effect the survey's response rate, but it will also improve the accuracy of the data collected. You want people who are interested in and knowledgeable (or familiar) with the survey's topic.

In many instances, it is not feasible to solicit every eligible respondent to take a survey; it may be too costly or involve too many people. You will want to employ a sampling method for such instance. We have a survey sampling document that explains several sampling methods.

Last updated: 2012 January 10