Simple Survey Idea 4: Don’t give the answers away

Do you ever “give away” answers in your surveys?  I’m talking about subtle (and not-so-subtle) signals that can lead to bias.  Here are a few errors to avoid:

Pandering

Several weeks ago I refinanced my house using an online lender.  All ended well, but there were a few glitches along the way – a key email with documents attached was apparently lost and I had to prompt the company to follow up with the underwriter.

The day after closing I received the following survey invitation from the mortgage processor:

Subject: I so appreciate YOU! Please help if you can I am so close to being # 1 in the company having “GREATs”…

Thank you so much for being an amazing customer to work with. I greatly appreciate all your help to get your loan taken care of. I hope that you feel I have given you “GREAT” customer service. My managers would love to hear what you think about my performance as your processor. If you do not mind, please take 1 minute to fill out the 1 question survey to help me out. We are always looking for “GREATs.”

Apparently customer-service ratings at that company are used in compensating or rewarding mortgage officers.  That’s fine.  But the question it raises is: Why would the company – which cares enough about satisfaction to tie it to rewards – let the person being evaluated pander for good ratings in the survey invitation?

You may have seen a more subtle form of this:

Thanks for coming to the SuperDuper Missions Conference.  Weren’t the speakers and worship music great?  Plus, over 300 people responded to the challenge to give or go.  Hopefully you were as blessed as I was.

Say, I would love to get your feedback to help us make future conferences even better!  Here’s a link to the survey…

It can be hard to contain enthusiasm when asking for post-event feedback – especially if you sent out several enthusiastic pre-event emails.  But if you want honest input, commit to avoiding remarks that suggest how the event should be evaluated (or how you would evaluate the event).

It Must Be Important Because They’re Asking About It

Most people have encountered surveys with leading questions, designed to confirm and publicize high levels of support for a position on an issue.  Like this:

Are you in favor of programs that offer microloans to lift women in developing countries out of the cycle of poverty with dignity through sustainable small businesses, with local peer-accountability networks to ensure loan repayment?

Even if you have read articles about recent studies suggesting that the link between microfinance and poverty reduction is tenuous or non-existent, you might be hard-pressed to answer “no” to the question as worded.

But there are other, more subtle ways that organizations can “suggest” certain responses.  Telling people in the survey invitation that the survey is about microloans can encourage people to overstate their interest in that topic (as well as leading to response bias in which interested people are more likely to respond at all).  Better to say that the survey is about strategies for poverty reduction or (broader still) addressing key areas of human need in the developing world.

This lets you gauge interest in your issue by mixing it in with several related issues, like this:

From the following list, please select up to three programs that you have been involved in, or would consider becoming involved in:

__ Well-digging programs to help provide a consistent healthy water supply

__ Community health education programs to teach villagers basic hygiene

__ Microloan programs to help women grow sustainable small businesses

__ Literacy programs to help kids and adults gain life and career skills

__ Legal advocacy and awareness to stem human trafficking

__ Theological education programs to equip first-generation church leaders

__ Sponsorship programs to sustain the education and nurture of at-risk kids

The rest of the survey can be about microloans.  But before tipping your hand, you learn about interest in that issue relative to other issues — and even the correlation of interest among issues.  Plus, you can use survey logic to excuse non-interested people from follow-up questions that don’t apply to them.

You can go even further to mask your interest in the survey issue, even while asking several questions specific to that issue.  Before starting the battery of questions about microloans, include a statement like this:

“Next, one of the above topic areas will be selected for a series of follow-up questions.”

The statement is truthful and adheres to research ethics — it does not say that the topic will be randomly selected. But it leaves open the possibility that those who sponsored the survey may be interested in several types of programs, not just microloans, encouraging greater honesty in responses.

Unnecessary Survey Branding

However, these approaches still won’t work if the survey invitation is sent from someone at “Microcredit Charitable Enterprises” and the survey is emblazoned with the charity’s logo.  There are many good reasons to brand a survey to your constituents, starting with an improved response rate.  But sometimes, branding can be counterproductive.

If objective input is key, consider using an outside research provider in order to avoid tipping your hand, especially since research ethics require researchers to identify themselves about who is collecting the data.

Allowing Everything to Be “Extremely Important”

Another way that researchers can “give away” answers is by letting people rate the importance of various items independently.  Take this question, for instance:

In selecting a child-sponsorship program, how important to you are the following items?  Please answer on a scale of 1 to 5, where 1 is “Not at All Important” and 5 is “Extremely Important”:

1    2    3    4    5   Sponsor’s ability to write to and visit the child

1    2    3    4    5   Receiving regular updates from the child

1    2    3    4    5   On-site monitoring of the child’s care/progress

1    2    3    4    5   Written policies regarding how children are selected

1    2    3    4    5   Annual reporting of how your money was used

All of those are important!  The question practically begs respondents to give each item a 5.  Will that information help the agency?  Maybe for external communication, but not in deciding which areas to promote or strengthen.

Instead, consider this alternative:

In selecting a child-sponsorship programs, how would you prioritize the following items?  Distribute a total of 100 points across the five items.

Or

Please order the following five elements of a child-sponsorship program according to their relative importance, from 1 “most important” to 5 “least important.”  You can use each number only once.

In most cases, relative-value questions will produce much more useful data.

Are there other ways that you have seen surveys “give away” answers to respondents?   Or avoid doing so?  Let us know about your experiences and ideas.

Simple Survey Idea #3: Give Something Away

When you do a survey, you are asking people for their time and their opinions.  People are increasingly aware of the value of both.

With that in mind, it is a good practice – even among those who already know and trust you – to give something away in appreciation for their input.  Doing so will bless people and build goodwill.  It will also improve your response rate (and therefore the quality of your data).  And it will make them more inclined to participate in future surveys.

“But we don’t have the budget to give anything away,” I sometimes hear people say.  I say, “If you can’t find something to give away, you’re not trying very hard.”  You don’t need budget – there are lots of ways to give survey responders something for free.

The first thing you should give them is a short survey.  That may be a topic for another Simple Survey Ideas post, but it’s so important that it always warrants mentioning.

You can give people access to the survey results, a good idea if your responders are peers/stakeholders and you know they will be interested in what you are learning.  Depending on your survey software, it may cost you some time to format and email results out to those who responded.  But if you use an online package like Survey Monkey, you can set up options to automatically show the survey results to date upon completion of the survey.  That “Instant Results” feature is even available on the free Survey Monkey package.

Of course, the first few people who respond won’t get a very complete picture, so you might also want to send people a link to the full set of responses once you complete and close the survey.  This option is available in all of Survey Monkey’s paid subscription plans.

Quick aside: the advantages that you get with the online services’ paid plans (unlimited responses, survey logic, ability to download data, HTTPS security) make them well worth the cost (vs. the free plan) for almost any survey.  Even if you are just doing a one-off survey, you should still sign up for a month and then cancel the subscription when you’re done.  Your survey is worth the $24 investment.

Another useful offering is free-information-of-interest-to-respondents.  I use this with virtually every survey I do.  You can almost always find an article or ebook or presentation or video related to your survey topic or to a common interest of the survey audience.  Even if you don’t produce content, you can always find something free on the Internet to direct people to.

In this way, you can say in your survey invitation, “Everyone who completes the survey will receive a free ‘Top 10′ list of resources about _____.”  It doesn’t matter that the list is out there on the Internet for anyone to find – linking people to it is delivering value.  With Survey Monkey, the option to redirect survey finishers to the website of your choice only comes with annual plans.  So, you may need the workaround of embedding your own link on the last page of the survey, so responders can get to your resource.  At the risk of going beyond “simple,” try something like this:

Thanks for completing our survey.  Before clicking “Done,” click <a href=”http://www.yoursite.org/” target=”_blank”>this link </a> to open a new window with the free resource we promised.

Be careful that the resource will be of interest to nearly everyone that you invite.  Giveaways that appeal only to a certain segment of your audience will lead to response bias.

Should you ask permission of the content provider in advance?  It’s a good idea but not required – groups that offer free content on the web typically want people to find that content.  You benefit them by linking to their site.  Groups that provide many free mission-related resources include the World Evangelical Alliance and the U.S. Center for World Mission.

A quick-response incentive promises resources to the first X number of responders.  This can be a good idea if you have a limited number of tangible resources to give away – and especially if you need responses quickly.

A related incentive is the sweepstakes prize offer, where respondents are randomly selected to receive a prize – usually something with significant value.  Many researchers use a combination of a free something-for-everyone resource with a high-value sweepstakes prize for a few randomly selected winners.

I like sweepstakes offers – they are fun and they work to generate response.  But you have to be responsible with them – some laws apply (see a quick overview here and know that this post does not constitute legal advice).  If you go this route, make sure that everyone who responds has an equal chance to win (even those who don’t meet the criteria for responding to your survey – nothing ruins a good survey like people lying to qualify for a prize), clearly communicate what and how many prizes you are giving away, eligibility, how and how often you can enter, when the giveaway will take place, how winners will be notified, approximate likelihood of winning, and any geographic or residency limitations.

That sounds like a lot, but consider that the following covers all of that without sounding too much like the legal disclaimer lingo in car dealer’s radio ad:

“You and up to 400 others who complete the survey by March 31 will qualify for a random prize drawing for one of 10 copies of the Operation World DVD.  One entry allowed per survey link.  In April GMI will inform the 10 winners by email – they will need a valid U.S. mailing address to receive their DVD.  Not valid where prohibited by law.”

How to manage a random drawing without hundreds of slips of paper and a huge hat?  Discover the RAND function in Excel.  Very handy – be sure to sort, save and print results for your records.

Also make sure to give away everything you promise.  If some people don’t claim their prize by a given date, move on to the next people on your randomized list.

Prize giveaways are appealing to most, but it is not unusual for those in ministry circles to steer clear of them because of their similarity to gambling games of chance.  Before launching a contest, be sure your organization’s leadership knows about it.  If you run into concerns, one alternative is to allow or encourage winners to donate their prize to charity.

Some survey sponsors use a charitable donation as the incentive itself, which carries real appeal for respondents.  One commercial firm I worked with leads off its surveys with a question like this:

In appreciation for your opinion, our firm will be distributing charitable donations totaling $1000.  From the following list, please select the charitable organization that you would like your portion of the donation to go toward:

__ Organization A

__ Organization B

etc.

If your group is a charitable organization, you can use a list of projects instead.  This works well if you can (truthfully) mention that an individual donor has put up the gift money to be distributed in this manner.

A final tip that applies to any gift or incentive that you offer: don’t position it as the primary reason to respond – especially in the subject line of an invitation email.  Not only do words like “prize” and “win” tend to trigger spam filters, but leading with the gift offer sends a message to invitees that you view the exercise as a transaction (or worse, that you think they are primarily motivated by greed).

Instead, keep the focus on the importance of the survey topic and the value of the person’s opinion – then mention the gift or prize.  As a survey sponsor, your identity should be that of a listener asking people for the favor of their input and offering them the opportunity for involvement – plus a gift as a token of your appreciation – rather than as a purchaser of people’s opinions.