Monthly Archives: March 2012

Onboarding: Needed for all Field Leader

1 Reply

Do you view onboarding as a necessary practice for first-time leaders of your teams, countries and areas?  If you are committed to the above perspective, that is a great beginning, but you are in a minority of mission leaders.  (If you are unfamiliar with onboarding, familiarize yourself with an introduction to the idea in the Feb. 15, 2012 post.)

The harsh reality is that most field leaders do not receive training or help regardless of whether they are first- time field leaders or promoted from another leadership capacity on the field.  Only 30% of field leaders, Continue reading →

Simple Survey Idea 4: Don’t give the answers away

1 Reply

Do you ever “give away” answers in your surveys?  I’m talking about subtle (and not-so-subtle) signals that can lead to bias.  Here are a few errors to avoid:

Pandering

Several weeks ago I refinanced my house using an online lender.  All ended well, but there were a few glitches along the way – a key email with documents attached was apparently lost and I had to prompt the company to follow up with the underwriter.

The day after closing I received the following survey invitation from the mortgage processor:

Subject: I so appreciate YOU! Please help if you can :) I am so close to being # 1 in the company having “GREATs”…

Thank you so much for being an amazing customer to work with. I greatly appreciate all your help to get your loan taken care of. I hope that you feel I have given you “GREAT” customer service. My managers would love to hear what you think about my performance as your processor. If you do not mind, please take 1 minute to fill out the 1 question survey to help me out. We are always looking for “GREATs.”

Apparently customer-service ratings at that company are used in compensating or rewarding mortgage officers.  That’s fine.  But the question it raises is: Why would the company – which cares enough about satisfaction to tie it to rewards – let the person being evaluated pander for good ratings in the survey invitation?

You may have seen a more subtle form of this:

Thanks for coming to the SuperDuper Missions Conference.  Weren’t the speakers and worship music great?  Plus, over 300 people responded to the challenge to give or go.  Hopefully you were as blessed as I was.

Say, I would love to get your feedback to help us make future conferences even better!  Here’s a link to the survey…

It can be hard to contain enthusiasm when asking for post-event feedback – especially if you sent out several enthusiastic pre-event emails.  But if you want honest input, commit to avoiding remarks that suggest how the event should be evaluated (or how you would evaluate the event).

It Must Be Important Because They’re Asking About It

Most people have encountered surveys with leading questions, designed to confirm and publicize high levels of support for a position on an issue.  Like this:

Are you in favor of programs that offer microloans to lift women in developing countries out of the cycle of poverty with dignity through sustainable small businesses, with local peer-accountability networks to ensure loan repayment?

Even if you have read articles about recent studies suggesting that the link between microfinance and poverty reduction is tenuous or non-existent, you might be hard-pressed to answer “no” to the question as worded.

But there are other, more subtle ways that organizations can “suggest” certain responses.  Telling people in the survey invitation that the survey is about microloans can encourage people to overstate their interest in that topic (as well as leading to response bias in which interested people are more likely to respond at all).  Better to say that the survey is about strategies for poverty reduction or (broader still) addressing key areas of human need in the developing world.

This lets you gauge interest in your issue by mixing it in with several related issues, like this:

From the following list, please select up to three programs that you have been involved in, or would consider becoming involved in:

__ Well-digging programs to help provide a consistent healthy water supply

__ Community health education programs to teach villagers basic hygiene

__ Microloan programs to help women grow sustainable small businesses

__ Literacy programs to help kids and adults gain life and career skills

__ Legal advocacy and awareness to stem human trafficking

__ Theological education programs to equip first-generation church leaders

__ Sponsorship programs to sustain the education and nurture of at-risk kids

The rest of the survey can be about microloans.  But before tipping your hand, you learn about interest in that issue relative to other issues — and even the correlation of interest among issues.  Plus, you can use survey logic to excuse non-interested people from follow-up questions that don’t apply to them.

You can go even further to mask your interest in the survey issue, even while asking several questions specific to that issue.  Before starting the battery of questions about microloans, include a statement like this:

“Next, one of the above topic areas will be selected for a series of follow-up questions.”

The statement is truthful and adheres to research ethics — it does not say that the topic will be randomly selected. But it leaves open the possibility that those who sponsored the survey may be interested in several types of programs, not just microloans, encouraging greater honesty in responses.

Unnecessary Survey Branding

However, these approaches still won’t work if the survey invitation is sent from someone at “Microcredit Charitable Enterprises” and the survey is emblazoned with the charity’s logo.  There are many good reasons to brand a survey to your constituents, starting with an improved response rate.  But sometimes, branding can be counterproductive.

If objective input is key, consider using an outside research provider in order to avoid tipping your hand, especially since research ethics require researchers to identify themselves about who is collecting the data.

Allowing Everything to Be “Extremely Important”

Another way that researchers can “give away” answers is by letting people rate the importance of various items independently.  Take this question, for instance:

In selecting a child-sponsorship program, how important to you are the following items?  Please answer on a scale of 1 to 5, where 1 is “Not at All Important” and 5 is “Extremely Important”:

1    2    3    4    5   Sponsor’s ability to write to and visit the child

1    2    3    4    5   Receiving regular updates from the child

1    2    3    4    5   On-site monitoring of the child’s care/progress

1    2    3    4    5   Written policies regarding how children are selected

1    2    3    4    5   Annual reporting of how your money was used

All of those are important!  The question practically begs respondents to give each item a 5.  Will that information help the agency?  Maybe for external communication, but not in deciding which areas to promote or strengthen.

Instead, consider this alternative:

In selecting a child-sponsorship programs, how would you prioritize the following items?  Distribute a total of 100 points across the five items.

Or

Please order the following five elements of a child-sponsorship program according to their relative importance, from 1 “most important” to 5 “least important.”  You can use each number only once.

In most cases, relative-value questions will produce much more useful data.

Are there other ways that you have seen surveys “give away” answers to respondents?   Or avoid doing so?  Let us know about your experiences and ideas.

Back to College – a week @biolau for #bmc2012

Leave a reply

I’ve been blessed to assist several student mission conferences with evaluations and planning research.  But until last week, I’d never actually attended one in person.  Not as a student.  Not as an adult.

So, my impressions of the Biola Missions Conference 2012 come unjaded – but also with no basis for comparison.  Take them as you will.

Here’s what stood out to me:

  1. The amazing level of creativity and production values of the young adults.  I have often heard people talk about the sophistication of today’s young people in regard to media.  They have highly developed filters, they are tough to impress, etc.  But that’s about their reaction to content produced by others.  Last week the content that students created themselves was on display.  And I was impressed.
    Dance, drama, visual art, poetry, film, music, oratory, cuisine, fashion – creativity seemed to be everywhere, all the time – and always expressing a passion for God and for mission.It’s one thing to attend a professionally produced plenary session with creative stage lighting, choreographed dance numbers, moving testimonies, etc.  But upon leaving the assembly hall, I walked past sidewalk artists in the process of creating scripture-based chalk drawings, then past elaborately decorated booths to advocate for and educate about peoples in unreached lands.  I caught the aroma of African food in the air as I walked to the end of campus.  There, the line stretched around the building for Global Awareness, a series of interactive role plays where participants might find themselves in the midst of a Somalian hostage crisis or a Chinese house church.

    Now, wowing a guy who barely falls on the fringe of Gen X may be a low bar.  So, check out these examples of their work and tell me what you think.

    Biola is close by Hollywood, and I met at least two students who hope to shine their light in the entertainment industry after college.  That may have had something to do with the emphasis on art and production.  But I’m still amazed that the whole thing was pulled together in the spare time of people taking a full load of classes.

  2. A student-run event has its glitches – but it also has great educational value.  I encountered half a dozen bumps that would never have happened at a professionally run event, but each of them represented a lesson in project management that won’t soon be forgotten.  And the event staff were accessible and eager to address issues when they arose.  They don’t deserve a complaining spirit from me.So, I’ll mention only one incident, and that just to share the smile it brought.  During the Welcome Tea for mission agency reps, a student leader said, “For those of you staying on campus, I hope you remembered to bring bedding.”  Note for next year’s conference: consider including that detail in the pre-event info packet.  As it turned out, my student host (and I think many others) provided sheets and pillows for their missionary guests, generously trading their beds for four nights on the floor.
  3. Staying in the dorms is a bit of an adventure for a 40-something guy, what with students’ odd hours, having to climb over a bookshelf to get into the loft, and sharing a bathroom with 30 people.  Still, I would do it again in a heartbeat – and not just to save money on a hotel room – because several of my best conversations with students took place there.  Definitely a shot-in-the-arm to interact with talented, God-fearing men with big dreams and passion for the Kingdom.

If any Biola students or others considering long-term cross-cultural service are reading this post, I’d be honored if you would be willing to give your opinions a few times a year on mission-related issues.  Sign up for our Mission Research Panel here.

It’s also a great time for college students who want to apply their skills, passion and creativity in mission research for a few months to apply for a GMI internship.  Send your resume to info@gmi.org.

Looking forward to attending and speaking at Biola again next year!

 

Simple Survey Idea #3: Give Something Away

Leave a reply

When you do a survey, you are asking people for their time and their opinions.  People are increasingly aware of the value of both.

With that in mind, it is a good practice – even among those who already know and trust you – to give something away in appreciation for their input.  Doing so will bless people and build goodwill.  It will also improve your response rate (and therefore the quality of your data).  And it will make them more inclined to participate in future surveys.

“But we don’t have the budget to give anything away,” I sometimes hear people say.  I say, “If you can’t find something to give away, you’re not trying very hard.”  You don’t need budget – there are lots of ways to give survey responders something for free.

The first thing you should give them is a short survey.  That may be a topic for another Simple Survey Ideas post, but it’s so important that it always warrants mentioning.

You can give people access to the survey results, a good idea if your responders are peers/stakeholders and you know they will be interested in what you are learning.  Depending on your survey software, it may cost you some time to format and email results out to those who responded.  But if you use an online package like Survey Monkey, you can set up options to automatically show the survey results to date upon completion of the survey.  That “Instant Results” feature is even available on the free Survey Monkey package.

Of course, the first few people who respond won’t get a very complete picture, so you might also want to send people a link to the full set of responses once you complete and close the survey.  This option is available in all of Survey Monkey’s paid subscription plans.

Quick aside: the advantages that you get with the online services’ paid plans (unlimited responses, survey logic, ability to download data, HTTPS security) make them well worth the cost (vs. the free plan) for almost any survey.  Even if you are just doing a one-off survey, you should still sign up for a month and then cancel the subscription when you’re done.  Your survey is worth the $24 investment.

Another useful offering is free-information-of-interest-to-respondents.  I use this with virtually every survey I do.  You can almost always find an article or ebook or presentation or video related to your survey topic or to a common interest of the survey audience.  Even if you don’t produce content, you can always find something free on the Internet to direct people to.

In this way, you can say in your survey invitation, “Everyone who completes the survey will receive a free ‘Top 10’ list of resources about _____.”  It doesn’t matter that the list is out there on the Internet for anyone to find – linking people to it is delivering value.  With Survey Monkey, the option to redirect survey finishers to the website of your choice only comes with annual plans.  So, you may need the workaround of embedding your own link on the last page of the survey, so responders can get to your resource.  At the risk of going beyond “simple,” try something like this:

Thanks for completing our survey.  Before clicking “Done,” click <a href=”http://www.yoursite.org/” target=”_blank”>this link </a> to open a new window with the free resource we promised.

Be careful that the resource will be of interest to nearly everyone that you invite.  Giveaways that appeal only to a certain segment of your audience will lead to response bias.

Should you ask permission of the content provider in advance?  It’s a good idea but not required – groups that offer free content on the web typically want people to find that content.  You benefit them by linking to their site.  Groups that provide many free mission-related resources include the World Evangelical Alliance and the U.S. Center for World Mission.

A quick-response incentive promises resources to the first X number of responders.  This can be a good idea if you have a limited number of tangible resources to give away – and especially if you need responses quickly.

A related incentive is the sweepstakes prize offer, where respondents are randomly selected to receive a prize – usually something with significant value.  Many researchers use a combination of a free something-for-everyone resource with a high-value sweepstakes prize for a few randomly selected winners.

I like sweepstakes offers – they are fun and they work to generate response.  But you have to be responsible with them – some laws apply (see a quick overview here and know that this post does not constitute legal advice).  If you go this route, make sure that everyone who responds has an equal chance to win (even those who don’t meet the criteria for responding to your survey – nothing ruins a good survey like people lying to qualify for a prize), clearly communicate what and how many prizes you are giving away, eligibility, how and how often you can enter, when the giveaway will take place, how winners will be notified, approximate likelihood of winning, and any geographic or residency limitations.

That sounds like a lot, but consider that the following covers all of that without sounding too much like the legal disclaimer lingo in car dealer’s radio ad:

“You and up to 400 others who complete the survey by March 31 will qualify for a random prize drawing for one of 10 copies of the Operation World DVD.  One entry allowed per survey link.  In April GMI will inform the 10 winners by email – they will need a valid U.S. mailing address to receive their DVD.  Not valid where prohibited by law.”

How to manage a random drawing without hundreds of slips of paper and a huge hat?  Discover the RAND function in Excel.  Very handy – be sure to sort, save and print results for your records.

Also make sure to give away everything you promise.  If some people don’t claim their prize by a given date, move on to the next people on your randomized list.

Prize giveaways are appealing to most, but it is not unusual for those in ministry circles to steer clear of them because of their similarity to gambling games of chance.  Before launching a contest, be sure your organization’s leadership knows about it.  If you run into concerns, one alternative is to allow or encourage winners to donate their prize to charity.

Some survey sponsors use a charitable donation as the incentive itself, which carries real appeal for respondents.  One commercial firm I worked with leads off its surveys with a question like this:

In appreciation for your opinion, our firm will be distributing charitable donations totaling $1000.  From the following list, please select the charitable organization that you would like your portion of the donation to go toward:

__ Organization A

__ Organization B

etc.

If your group is a charitable organization, you can use a list of projects instead.  This works well if you can (truthfully) mention that an individual donor has put up the gift money to be distributed in this manner.

A final tip that applies to any gift or incentive that you offer: don’t position it as the primary reason to respond – especially in the subject line of an invitation email.  Not only do words like “prize” and “win” tend to trigger spam filters, but leading with the gift offer sends a message to invitees that you view the exercise as a transaction (or worse, that you think they are primarily motivated by greed).

Instead, keep the focus on the importance of the survey topic and the value of the person’s opinion – then mention the gift or prize.  As a survey sponsor, your identity should be that of a listener asking people for the favor of their input and offering them the opportunity for involvement – plus a gift as a token of your appreciation – rather than as a purchaser of people’s opinions.

 

Onboarding: What is the driving thrust?

3 Replies

Helping another person move into a new position is often thought of as primarily a training function.  If we can only get the new person to have the right knowledge, they can get on with the job.   Thus, orientation programs in most organizations are times to give input on organizational policies/procedures and a few basic skills.

While effective onboarding will use training and input, that effort is only the beginning.  Continue reading →

Simple Survey Idea #2: Send a Reminder

Leave a reply

I talk with lots of people who design and field their own web surveys.  It amazes me how many have never considered sending a reminder out to those they have invited — even to people who are known well by the person doing the survey.

People are often very willing to help, but they are busy and working through lots of messages, and survey invitations are easy to set aside until later.  One reminder is often helpful.  I almost always send at least one reminder out to survey invitees.  In some cases, I will send out a second reminder.  In rare cases, a third.

Why send a reminder at all?  Perhaps it goes without saying, but more data usually equals better-quality information.  Better statistical accuracy is part of that: most people understand that a sample of 300 yields a tighter margin of error than a sample of 100.

But in most cases, response bias will be a bigger threat to the quality of your data than statistical error from sample size.  Consider your sample of 300 responses.  Did you generate those from 400 invitations (a 75% response rate) or 4,000 invitations (a 7.5% response rate)?  The former would give you much greater confidence that those you heard from accurately reflect the larger group that you invited.

What is a “good” response rate?  It can vary widely depending on your relationship to the people invited (as well as how interesting and long the survey is, but that’s a topic for another post).  Domestic staff/employee surveys often generate a response of 85 percent or more.  However, for internationally distributed missionary staff, a response of 60 percent is healthy.  For audiences with an established interest in your work (event attenders, network members), a 35-percent response is decent.  For other audiences, expect something lower.  One online survey supplier’s analysis of nearly 200 surveys indicated a median respose rate of 26 percent.

So, do reminders substantially increase response to surveys?  Absolutely.  Online survey provider Vovici blogs, “Following up survey invitations with reminders is the most dramatic way to improve your response rate.”  They show results from one survey where the response rate rose from 14 percent to 23, 28 and 33 percent after subsequent reminders.

My experience has been similar.  I find that survey invitations and reminders have something like a “half-life” effect.  If your initial invitation generates X responses, you can expect a first reminder to produce an additional .50X responses, a second reminder .25X responses, and so on.

I disagree with survey provider Zoomerang’s suggestion of sending a series of three reminders — especially if the audience is people you know — but I do agree with their statement, “Think of your first email reminder as a favor, not an annoyance.”  I recommend sending at least one reminder for virtually any survey, with a second reminder only if you feel that your response rate is troublesome and you need that extra .25X of input.

At least Zoomerang provides a sample reminder template you can use.  I agree that you should keep reminders short — shorter than the original invitation.  With any invitation or reminder, you will do well to keep the survey link “above the fold” (to use a phrase from old-time print journalism), meaning that it should be visible to readers without their having to scroll down through your message.

I also find that it very helpful to use list managers in sending survey reminders.  Most online providers will have an option where you can send only to those members of your invitation list who haven’t responded.  Not only does this keep from annoying those who already did respond, but you can word the reminder much more directly (and personally, with customized name fields).  So, instead of saying:

“Dear friend — If you haven’t already responded to our survey, please do so today.”

You can say:

“Dear Zach — I notice that you haven’t responded to our survey yet.  No problem, I’m sure you’re busy.  But it would be great to get your input today.  Here’s the link.”

Take care in using the above approach — if you have promised anonymity (not just confidentiality), as in an employee survey, opt for the generic reminder.

When to send a reminder?  If your schedule is not pressing, send a reminder out 5-10 days after the previous contact.  I recommend varying the time of day and week in order to connect with different kinds of people.  If I sent the initial invitation on a Monday morning, I might send the reminder the following Wednesday afternoon.