Back to College – a week @biolau for #bmc2012

I’ve been blessed to assist several student mission conferences with evaluations and planning research.  But until last week, I’d never actually attended one in person.  Not as a student.  Not as an adult.

So, my impressions of the Biola Missions Conference 2012 come unjaded – but also with no basis for comparison.  Take them as you will.

Here’s what stood out to me:

  1. The amazing level of creativity and production values of the young adults.  I have often heard people talk about the sophistication of today’s young people in regard to media.  They have highly developed filters, they are tough to impress, etc.  But that’s about their reaction to content produced by others.  Last week the content that students created themselves was on display.  And I was impressed.
    Dance, drama, visual art, poetry, film, music, oratory, cuisine, fashion – creativity seemed to be everywhere, all the time – and always expressing a passion for God and for mission.It’s one thing to attend a professionally produced plenary session with creative stage lighting, choreographed dance numbers, moving testimonies, etc.  But upon leaving the assembly hall, I walked past sidewalk artists in the process of creating scripture-based chalk drawings, then past elaborately decorated booths to advocate for and educate about peoples in unreached lands.  I caught the aroma of African food in the air as I walked to the end of campus.  There, the line stretched around the building for Global Awareness, a series of interactive role plays where participants might find themselves in the midst of a Somalian hostage crisis or a Chinese house church.

    Now, wowing a guy who barely falls on the fringe of Gen X may be a low bar.  So, check out these examples of their work and tell me what you think.

    Biola is close by Hollywood, and I met at least two students who hope to shine their light in the entertainment industry after college.  That may have had something to do with the emphasis on art and production.  But I’m still amazed that the whole thing was pulled together in the spare time of people taking a full load of classes.

  2. A student-run event has its glitches – but it also has great educational value.  I encountered half a dozen bumps that would never have happened at a professionally run event, but each of them represented a lesson in project management that won’t soon be forgotten.  And the event staff were accessible and eager to address issues when they arose.  They don’t deserve a complaining spirit from me.So, I’ll mention only one incident, and that just to share the smile it brought.  During the Welcome Tea for mission agency reps, a student leader said, “For those of you staying on campus, I hope you remembered to bring bedding.”  Note for next year’s conference: consider including that detail in the pre-event info packet.  As it turned out, my student host (and I think many others) provided sheets and pillows for their missionary guests, generously trading their beds for four nights on the floor.
  3. Staying in the dorms is a bit of an adventure for a 40-something guy, what with students’ odd hours, having to climb over a bookshelf to get into the loft, and sharing a bathroom with 30 people.  Still, I would do it again in a heartbeat – and not just to save money on a hotel room – because several of my best conversations with students took place there.  Definitely a shot-in-the-arm to interact with talented, God-fearing men with big dreams and passion for the Kingdom.

If any Biola students or others considering long-term cross-cultural service are reading this post, I’d be honored if you would be willing to give your opinions a few times a year on mission-related issues.  Sign up for our Mission Research Panel here.

It’s also a great time for college students who want to apply their skills, passion and creativity in mission research for a few months to apply for a GMI internship.  Send your resume to info@gmi.org.

Looking forward to attending and speaking at Biola again next year!

 

Simple Survey Idea #3: Give Something Away

When you do a survey, you are asking people for their time and their opinions.  People are increasingly aware of the value of both.

With that in mind, it is a good practice – even among those who already know and trust you – to give something away in appreciation for their input.  Doing so will bless people and build goodwill.  It will also improve your response rate (and therefore the quality of your data).  And it will make them more inclined to participate in future surveys.

“But we don’t have the budget to give anything away,” I sometimes hear people say.  I say, “If you can’t find something to give away, you’re not trying very hard.”  You don’t need budget – there are lots of ways to give survey responders something for free.

The first thing you should give them is a short survey.  That may be a topic for another Simple Survey Ideas post, but it’s so important that it always warrants mentioning.

You can give people access to the survey results, a good idea if your responders are peers/stakeholders and you know they will be interested in what you are learning.  Depending on your survey software, it may cost you some time to format and email results out to those who responded.  But if you use an online package like Survey Monkey, you can set up options to automatically show the survey results to date upon completion of the survey.  That “Instant Results” feature is even available on the free Survey Monkey package.

Of course, the first few people who respond won’t get a very complete picture, so you might also want to send people a link to the full set of responses once you complete and close the survey.  This option is available in all of Survey Monkey’s paid subscription plans.

Quick aside: the advantages that you get with the online services’ paid plans (unlimited responses, survey logic, ability to download data, HTTPS security) make them well worth the cost (vs. the free plan) for almost any survey.  Even if you are just doing a one-off survey, you should still sign up for a month and then cancel the subscription when you’re done.  Your survey is worth the $24 investment.

Another useful offering is free-information-of-interest-to-respondents.  I use this with virtually every survey I do.  You can almost always find an article or ebook or presentation or video related to your survey topic or to a common interest of the survey audience.  Even if you don’t produce content, you can always find something free on the Internet to direct people to.

In this way, you can say in your survey invitation, “Everyone who completes the survey will receive a free ‘Top 10’ list of resources about _____.”  It doesn’t matter that the list is out there on the Internet for anyone to find – linking people to it is delivering value.  With Survey Monkey, the option to redirect survey finishers to the website of your choice only comes with annual plans.  So, you may need the workaround of embedding your own link on the last page of the survey, so responders can get to your resource.  At the risk of going beyond “simple,” try something like this:

Thanks for completing our survey.  Before clicking “Done,” click <a href=”http://www.yoursite.org/” target=”_blank”>this link </a> to open a new window with the free resource we promised.

Be careful that the resource will be of interest to nearly everyone that you invite.  Giveaways that appeal only to a certain segment of your audience will lead to response bias.

Should you ask permission of the content provider in advance?  It’s a good idea but not required – groups that offer free content on the web typically want people to find that content.  You benefit them by linking to their site.  Groups that provide many free mission-related resources include the World Evangelical Alliance and the U.S. Center for World Mission.

A quick-response incentive promises resources to the first X number of responders.  This can be a good idea if you have a limited number of tangible resources to give away – and especially if you need responses quickly.

A related incentive is the sweepstakes prize offer, where respondents are randomly selected to receive a prize – usually something with significant value.  Many researchers use a combination of a free something-for-everyone resource with a high-value sweepstakes prize for a few randomly selected winners.

I like sweepstakes offers – they are fun and they work to generate response.  But you have to be responsible with them – some laws apply (see a quick overview here and know that this post does not constitute legal advice).  If you go this route, make sure that everyone who responds has an equal chance to win (even those who don’t meet the criteria for responding to your survey – nothing ruins a good survey like people lying to qualify for a prize), clearly communicate what and how many prizes you are giving away, eligibility, how and how often you can enter, when the giveaway will take place, how winners will be notified, approximate likelihood of winning, and any geographic or residency limitations.

That sounds like a lot, but consider that the following covers all of that without sounding too much like the legal disclaimer lingo in car dealer’s radio ad:

“You and up to 400 others who complete the survey by March 31 will qualify for a random prize drawing for one of 10 copies of the Operation World DVD.  One entry allowed per survey link.  In April GMI will inform the 10 winners by email – they will need a valid U.S. mailing address to receive their DVD.  Not valid where prohibited by law.”

How to manage a random drawing without hundreds of slips of paper and a huge hat?  Discover the RAND function in Excel.  Very handy – be sure to sort, save and print results for your records.

Also make sure to give away everything you promise.  If some people don’t claim their prize by a given date, move on to the next people on your randomized list.

Prize giveaways are appealing to most, but it is not unusual for those in ministry circles to steer clear of them because of their similarity to gambling games of chance.  Before launching a contest, be sure your organization’s leadership knows about it.  If you run into concerns, one alternative is to allow or encourage winners to donate their prize to charity.

Some survey sponsors use a charitable donation as the incentive itself, which carries real appeal for respondents.  One commercial firm I worked with leads off its surveys with a question like this:

In appreciation for your opinion, our firm will be distributing charitable donations totaling $1000.  From the following list, please select the charitable organization that you would like your portion of the donation to go toward:

__ Organization A

__ Organization B

etc.

If your group is a charitable organization, you can use a list of projects instead.  This works well if you can (truthfully) mention that an individual donor has put up the gift money to be distributed in this manner.

A final tip that applies to any gift or incentive that you offer: don’t position it as the primary reason to respond – especially in the subject line of an invitation email.  Not only do words like “prize” and “win” tend to trigger spam filters, but leading with the gift offer sends a message to invitees that you view the exercise as a transaction (or worse, that you think they are primarily motivated by greed).

Instead, keep the focus on the importance of the survey topic and the value of the person’s opinion – then mention the gift or prize.  As a survey sponsor, your identity should be that of a listener asking people for the favor of their input and offering them the opportunity for involvement – plus a gift as a token of your appreciation – rather than as a purchaser of people’s opinions.

 

Onboarding: What is the driving thrust?

Helping another person move into a new position is often thought of as primarily a training function.  If we can only get the new person to have the right knowledge, they can get on with the job.   Thus, orientation programs in most organizations are times to give input on organizational policies/procedures and a few basic skills.

While effective onboarding will use training and input, that effort is only the beginning.  Continue reading

Simple Survey Idea #2: Send a Reminder

I talk with lots of people who design and field their own web surveys.  It amazes me how many have never considered sending a reminder out to those they have invited — even to people who are known well by the person doing the survey.

People are often very willing to help, but they are busy and working through lots of messages, and survey invitations are easy to set aside until later.  One reminder is often helpful.  I almost always send at least one reminder out to survey invitees.  In some cases, I will send out a second reminder.  In rare cases, a third.

Why send a reminder at all?  Perhaps it goes without saying, but more data usually equals better-quality information.  Better statistical accuracy is part of that: most people understand that a sample of 300 yields a tighter margin of error than a sample of 100.

But in most cases, response bias will be a bigger threat to the quality of your data than statistical error from sample size.  Consider your sample of 300 responses.  Did you generate those from 400 invitations (a 75% response rate) or 4,000 invitations (a 7.5% response rate)?  The former would give you much greater confidence that those you heard from accurately reflect the larger group that you invited.

What is a “good” response rate?  It can vary widely depending on your relationship to the people invited (as well as how interesting and long the survey is, but that’s a topic for another post).  Domestic staff/employee surveys often generate a response of 85 percent or more.  However, for internationally distributed missionary staff, a response of 60 percent is healthy.  For audiences with an established interest in your work (event attenders, network members), a 35-percent response is decent.  For other audiences, expect something lower.  One online survey supplier’s analysis of nearly 200 surveys indicated a median respose rate of 26 percent.

So, do reminders substantially increase response to surveys?  Absolutely.  Online survey provider Vovici blogs, “Following up survey invitations with reminders is the most dramatic way to improve your response rate.”  They show results from one survey where the response rate rose from 14 percent to 23, 28 and 33 percent after subsequent reminders.

My experience has been similar.  I find that survey invitations and reminders have something like a “half-life” effect.  If your initial invitation generates X responses, you can expect a first reminder to produce an additional .50X responses, a second reminder .25X responses, and so on.

I disagree with survey provider Zoomerang’s suggestion of sending a series of three reminders — especially if the audience is people you know — but I do agree with their statement, “Think of your first email reminder as a favor, not an annoyance.”  I recommend sending at least one reminder for virtually any survey, with a second reminder only if you feel that your response rate is troublesome and you need that extra .25X of input.

At least Zoomerang provides a sample reminder template you can use.  I agree that you should keep reminders short — shorter than the original invitation.  With any invitation or reminder, you will do well to keep the survey link “above the fold” (to use a phrase from old-time print journalism), meaning that it should be visible to readers without their having to scroll down through your message.

I also find that it very helpful to use list managers in sending survey reminders.  Most online providers will have an option where you can send only to those members of your invitation list who haven’t responded.  Not only does this keep from annoying those who already did respond, but you can word the reminder much more directly (and personally, with customized name fields).  So, instead of saying:

“Dear friend — If you haven’t already responded to our survey, please do so today.”

You can say:

“Dear Zach — I notice that you haven’t responded to our survey yet.  No problem, I’m sure you’re busy.  But it would be great to get your input today.  Here’s the link.”

Take care in using the above approach — if you have promised anonymity (not just confidentiality), as in an employee survey, opt for the generic reminder.

When to send a reminder?  If your schedule is not pressing, send a reminder out 5-10 days after the previous contact.  I recommend varying the time of day and week in order to connect with different kinds of people.  If I sent the initial invitation on a Monday morning, I might send the reminder the following Wednesday afternoon.

 

Simple Survey Idea #1: Keep survey language simple

I am working on a web survey for a group of people in India.  Smart folks, many of them technology savvy.  And they speak English — but often not as their first language.

Some surveys should be translated or fielded in multiple languages.  For many surveys, though, English is sufficient.  But what kind of English?

My default mode is to use more complicated English than is needed.  The more I work with multilingual people around the world, the more I realize the value of keeping language simple, especially with surveys and interview questions.

The good news is that there are tools out there that can help.  Here is a site that lets you paste in text and compare it to one of many collections of simple English words.  It shows which words are not considered simple.

http://www.online-utility.org/english/simple_basic_helper.jsp 

With many international audiences it is a good idea to test your language before sending out a survey.

I put the above portion of this post into the site and found out that the following words were not included in a somewhat large collection of 15,000 simple words: web, savvy, and multilingual.  With smaller collections, many more words miss the cut, including realizecompare and survey.

Try it out — you have nothing to lose but complexity.

Coaching: What really motivates people?

Coaching is a growing trend within mission organizations.  Coaching is not only a good practice for missionary care staff, consultants or external coaches.  It is also a needed competence for all mission leaders, including team, country and area/regional leaders (e.g. field leaders).

Coaching has obvious benefits in working with missionaries.  It is also an effective tool in the oversight and development of field leaders.  It is also an essential tool for the onboarding of new field leaders

Here are some questions about coaching impact: Continue reading

It’s Winter: Time to Make Snowballs!

A lot of mission researchers are interested in studying people who aren’t easy to get to.  They may be unknown in number, difficult to access, suspicious of outsiders, etc.

This makes random sampling virtually impossible.  Unfortunately, a random sample is an assumption or requirement of many statistical tests.

So, if you’re doing research with underground believers or people exploited in human trafficking, you can’t just go to SSI and rent a sample of 1500 people to call or email.

When you need a sample from a hard-to-reach population, make a snowball!

Snowball sampling, a more memorable name for the formal term, respondent-driven sampling, is a means of getting to a reasonably large sample through referrals.  You find some people who meet your criteria and who trust you enough to answer your questions, then ask them if there are other people like them that they could introduce you to.

In each interview, you ask for referrals – and pretty soon the snowball effect kicks in and you have a large sample.

For years this approach was avoided by “serious” researchers because, well, the sample it produces just isn’t random.  Your friends are probably more like you than the average person, so talking to you and your friends isn’t a great way to get a handle on your community.

But, like six-degrees of separation, the further you go from your original “seeds,” the broader the perspective.  And in recent years, formulas have been developed that virtually remove the bias inherent in snowball samples – opening up this method to “respectable” researchers.

How to do it?  Some researchers simply throw out the first two or three generations of data, then keep everything else, relying on three degrees of separation.  Not a bad rule of thumb.

For more serious researchers, there is free software available to help you weight the data and prevent you from having to discard the input of the nice people who got your snowball started.  Douglas Heckathorn is a Cornell professor who developed the algorithm (while doing research among drug users to help combat the spread of HIV) and helped bring snowball sampling back from the hinterlands of researcher scorn.  You can read more about his method here and download the software here.

Suddenly, you need not settle for a handful of isolated snowflakes, nor for a skewed snowdrift of opinion (via an unscientific poll of your social media friends).  Instead, you can craft their referrals into a statistically representative snowman.

Meanwhile, if the sample you need is one of North American field missionaries or North Americans seriously considering long-term cross-cultural service, you should consider renting one of GMI’s mission research panels.  Email us for details.

Onboarding: An Introduction

In this blog we take the general concepts of onboarding and apply them to the roles of team, country, and area/regional leaders within mission organizations.

1.     What is onboarding?
Onboarding is an intentional process to help new field leaders become quickly effective in the basics in their new roles.  It is a practical process to help a new field leader get “on board” in the new role promptly.  For a general introduction to onboarding in the business world, read Workforce Management Online, April 2011.

2.     Why is it needed?
The need for onboarding of new field leaders is imperative for at least two reasons, as discovered in IMPACT research of 277 field leaders.*  Continue reading

Analyzing open-ended questions: A bright future for word clouds

Commercial survey research firms usually charge clients significantly extra to include “open-ended” questions in a survey.  They tend to be messy and time-consuming.  Traditionally, analysts would read through a selection of responses to create categories of frequent or typical responses, then read back through all of the responses to categorize them.

For publishing in a peer-reviewed journal, multiple people would categorize responses independently, then work together to create a synthesized coding scheme to limit bias.

Most qualitative text-analysis software still requires you to manually “code” responses.

With all that work, open-ended questions are still important in exploratory and qualitative research – and frequently satisfying for survey respondents looking for an opportunity to say what is on their mind, unhindered by structured response categories.

But the tag-cloud age has been a blessing to those without the time and money to do full, traditional analysis of text responses.   Graphics with words sized by frequency of use enables analysts to quickly get a sense of the nature of open-ended responses.

New editions of survey software – even budget packages like Survey Monkey – include cloud-creating tools to help users understand open-ended responses at a glance, without all the coding work.

Even those doing traditional coding enjoy working with clouds, which help analysts to quickly create an initial set of codes.

If your survey package doesn’t have cloud-generating capacity, no problem.  Worldle is a free site that lets you create art-like word clouds.  The clouds in the previous post were created using Worldle.  It’s a terrific, easy-to-use site that lets you paste in your text – our data came straight from a spreadsheet – and generate a word cloud with one click.  It automatically removes common words, allows you to choose the relative cloud shape, color scheme, font and orientation of the words.  We chose to illustrate the top 100 terms for each question.  Wordle lets you save and use your clouds however you want to.

I really like the tool’s artistic quality.  Wordle clouds almost beg to be shown to others.  Then they become motivated, too.  My daughter, upon first seeing Wordle, immediately had a vision about making a sign to promote a bake sale.  A few descriptive terms later, she had created a beautiful graphic to draw people’s attention.

This is where research moves from information to influence.  Imagine asking your constituents about their needs – or your organization’s impact – then printing a graphic of their responses to hang in your office as a reminder and motivator to staff.  Unlike a research report, which may or may not get read before being filed away (or worse!), word cloud art can keep research right in front of your team.  The graphic format makes the information more memorable as well.

Researchers, meanwhile, can compare and contrast different audience segments, as I did in the word cloud below.

What applications can you think of for word clouds?

New research: Perceptions about mission internships

This post teases GMI’s upcoming survey report on perceptions of mission internships among current and future missionaries.

UK-based mission researcher and Redcliffe College principal Rob Hay told me (years ago) that his research among future missionaries in Europe showed that many intended to serve long term, but few were willing to make a career-length commitment to an individual agency.

So, GMI thought that mission internships (6 months to 3 years) would make a good survey topic for its North American panels of long-term cross-cultural field missionaries and people considering long-term cross-cultural field service.

This post deals with an open-ended question about perceptions.  More than 300 field missionaries and more than 300 people considering long-term cross-cultural service were asked:

When you think of cross-cultural field internships, what are a few of the first words that come to mind?

Here are word clouds reflecting the frequency of the top 100 response terms.

Prospective Cross-Cultural Missionaries:

Current Cross-Cultural Missionaries: 

What do you notice about these two word clouds?  Here are a few things I noticed:

  • For future missionaries, internships appear to be a learning experience that comes with a healthy dose of work.  For current field missionaries, internships appear to be a work experience that comes with a healthy dose of learning.
  • Future missionaries see internships as a spiritual experience. God was prospective interns’ fourth-most-frequently used term.  Also, Jesus, prayer and faith appear among frequently-used terms.  This group is eager to see God and to be a part of God’s work in the world.
  • By contrast, current field staff view internships as a ministry experience, something related to work and profession.  You can see Godin the word cloud, but on a much smaller scale, and almost no other spiritual terms.  Hopefully, this isn’t reflective of the spirituality of field missionaries!  Rather, it shows that they view internships through the lens of work activity.
  • Work dominates the current field missionary cloud.  They likely view internships not only as a lot of work for the intern, but also as a lot of work for resident missionaries.  Interns require a good deal of supervision and mentoring, which takes time.  There are immediate benefits in terms of enthusiastic service and future benefits in terms of long-term recruiting potential, but obtaining those rewards has a cost: work.
  • Both groups see experience as a key descriptor of mission internships.  But in what sense is the word used?  Is it about having a valuable experience (to be reflected upon) or is it about gaining experience (to be applied in the future)?  For current missionaries, the perspective is forward-looking, with future, preparation and potential appearing.  For prospects, there is a bit more balance.  While long-term and training are prominent, so is rewarding – suggesting intrinsic value, not just preparatory value.
  • Both groups see internships as an opportunity and are generally positive about them.  Good and great appear in moderate-to-large type in both clouds.  Current field workers also used the terms beneficial, excellent, positive and effective.
  • Both groups also are well aware that mission service of any length is challenging.
  • Time is a key element in internships.  In both clouds, the words time, term, long and short appear.
  • Financial considerations are an issue for prospective missionaries.  Support and fundraising aren’t the most prominent terms, but they are a real challenge/barrier for some prospects.  Those terms don’t show up on field missionaries’ radar as a key element of internships – they may assume that people considering long-term service will certainly be able to raise funding for a shorter-term opportunity (with those dollars easier to raise because they do not require a long-term commitment from donors).

What do you see in these clouds?  What are you concerned that you can’t see?