Monthly Archives: February 2012

Simple Survey Idea #1: Keep survey language simple

I am working on a web survey for a group of people in India.  Smart folks, many of them technology savvy.  And they speak English — but often not as their first language.

Some surveys should be translated or fielded in multiple languages.  For many surveys, though, English is sufficient.  But what kind of English?

My default mode is to use more complicated English than is needed.  The more I work with multilingual people around the world, the more I realize the value of keeping language simple, especially with surveys and interview questions.

The good news is that there are tools out there that can help.  Here is a site that lets you paste in text and compare it to one of many collections of simple English words.  It shows which words are not considered simple.

http://www.online-utility.org/english/simple_basic_helper.jsp 

With many international audiences it is a good idea to test your language before sending out a survey.

I put the above portion of this post into the site and found out that the following words were not included in a somewhat large collection of 15,000 simple words: web, savvy, and multilingual.  With smaller collections, many more words miss the cut, including realizecompare and survey.

Try it out — you have nothing to lose but complexity.

Coaching: What really motivates people?

Coaching is a growing trend within mission organizations.  Coaching is not only a good practice for missionary care staff, consultants or external coaches.  It is also a needed competence for all mission leaders, including team, country and area/regional leaders (e.g. field leaders).

Coaching has obvious benefits in working with missionaries.  It is also an effective tool in the oversight and development of field leaders.  It is also an essential tool for the onboarding of new field leaders

Here are some questions about coaching impact: Continue reading

It’s Winter: Time to Make Snowballs!

A lot of mission researchers are interested in studying people who aren’t easy to get to.  They may be unknown in number, difficult to access, suspicious of outsiders, etc.

This makes random sampling virtually impossible.  Unfortunately, a random sample is an assumption or requirement of many statistical tests.

So, if you’re doing research with underground believers or people exploited in human trafficking, you can’t just go to SSI and rent a sample of 1500 people to call or email.

When you need a sample from a hard-to-reach population, make a snowball!

Snowball sampling, a more memorable name for the formal term, respondent-driven sampling, is a means of getting to a reasonably large sample through referrals.  You find some people who meet your criteria and who trust you enough to answer your questions, then ask them if there are other people like them that they could introduce you to.

In each interview, you ask for referrals – and pretty soon the snowball effect kicks in and you have a large sample.

For years this approach was avoided by “serious” researchers because, well, the sample it produces just isn’t random.  Your friends are probably more like you than the average person, so talking to you and your friends isn’t a great way to get a handle on your community.

But, like six-degrees of separation, the further you go from your original “seeds,” the broader the perspective.  And in recent years, formulas have been developed that virtually remove the bias inherent in snowball samples – opening up this method to “respectable” researchers.

How to do it?  Some researchers simply throw out the first two or three generations of data, then keep everything else, relying on three degrees of separation.  Not a bad rule of thumb.

For more serious researchers, there is free software available to help you weight the data and prevent you from having to discard the input of the nice people who got your snowball started.  Douglas Heckathorn is a Cornell professor who developed the algorithm (while doing research among drug users to help combat the spread of HIV) and helped bring snowball sampling back from the hinterlands of researcher scorn.  You can read more about his method here and download the software here.

Suddenly, you need not settle for a handful of isolated snowflakes, nor for a skewed snowdrift of opinion (via an unscientific poll of your social media friends).  Instead, you can craft their referrals into a statistically representative snowman.

Meanwhile, if the sample you need is one of North American field missionaries or North Americans seriously considering long-term cross-cultural service, you should consider renting one of GMI’s mission research panels.  Email us for details.

Onboarding: An Introduction

In this blog we take the general concepts of onboarding and apply them to the roles of team, country, and area/regional leaders within mission organizations.

1.     What is onboarding?
Onboarding is an intentional process to help new field leaders become quickly effective in the basics in their new roles.  It is a practical process to help a new field leader get “on board” in the new role promptly.  For a general introduction to onboarding in the business world, read Workforce Management Online, April 2011.

2.     Why is it needed?
The need for onboarding of new field leaders is imperative for at least two reasons, as discovered in IMPACT research of 277 field leaders.*  Continue reading

Analyzing open-ended questions: A bright future for word clouds

Commercial survey research firms usually charge clients significantly extra to include “open-ended” questions in a survey.  They tend to be messy and time-consuming.  Traditionally, analysts would read through a selection of responses to create categories of frequent or typical responses, then read back through all of the responses to categorize them.

For publishing in a peer-reviewed journal, multiple people would categorize responses independently, then work together to create a synthesized coding scheme to limit bias.

Most qualitative text-analysis software still requires you to manually “code” responses.

With all that work, open-ended questions are still important in exploratory and qualitative research – and frequently satisfying for survey respondents looking for an opportunity to say what is on their mind, unhindered by structured response categories.

But the tag-cloud age has been a blessing to those without the time and money to do full, traditional analysis of text responses.   Graphics with words sized by frequency of use enables analysts to quickly get a sense of the nature of open-ended responses.

New editions of survey software – even budget packages like Survey Monkey – include cloud-creating tools to help users understand open-ended responses at a glance, without all the coding work.

Even those doing traditional coding enjoy working with clouds, which help analysts to quickly create an initial set of codes.

If your survey package doesn’t have cloud-generating capacity, no problem.  Worldle is a free site that lets you create art-like word clouds.  The clouds in the previous post were created using Worldle.  It’s a terrific, easy-to-use site that lets you paste in your text – our data came straight from a spreadsheet – and generate a word cloud with one click.  It automatically removes common words, allows you to choose the relative cloud shape, color scheme, font and orientation of the words.  We chose to illustrate the top 100 terms for each question.  Wordle lets you save and use your clouds however you want to.

I really like the tool’s artistic quality.  Wordle clouds almost beg to be shown to others.  Then they become motivated, too.  My daughter, upon first seeing Wordle, immediately had a vision about making a sign to promote a bake sale.  A few descriptive terms later, she had created a beautiful graphic to draw people’s attention.

This is where research moves from information to influence.  Imagine asking your constituents about their needs – or your organization’s impact – then printing a graphic of their responses to hang in your office as a reminder and motivator to staff.  Unlike a research report, which may or may not get read before being filed away (or worse!), word cloud art can keep research right in front of your team.  The graphic format makes the information more memorable as well.

Researchers, meanwhile, can compare and contrast different audience segments, as I did in the word cloud below.

What applications can you think of for word clouds?

New research: Perceptions about mission internships

This post teases GMI’s upcoming survey report on perceptions of mission internships among current and future missionaries.

UK-based mission researcher and Redcliffe College principal Rob Hay told me (years ago) that his research among future missionaries in Europe showed that many intended to serve long term, but few were willing to make a career-length commitment to an individual agency.

So, GMI thought that mission internships (6 months to 3 years) would make a good survey topic for its North American panels of long-term cross-cultural field missionaries and people considering long-term cross-cultural field service.

This post deals with an open-ended question about perceptions.  More than 300 field missionaries and more than 300 people considering long-term cross-cultural service were asked:

When you think of cross-cultural field internships, what are a few of the first words that come to mind?

Here are word clouds reflecting the frequency of the top 100 response terms.

Prospective Cross-Cultural Missionaries:

Current Cross-Cultural Missionaries: 

What do you notice about these two word clouds?  Here are a few things I noticed:

  • For future missionaries, internships appear to be a learning experience that comes with a healthy dose of work.  For current field missionaries, internships appear to be a work experience that comes with a healthy dose of learning.
  • Future missionaries see internships as a spiritual experience. God was prospective interns’ fourth-most-frequently used term.  Also, Jesus, prayer and faith appear among frequently-used terms.  This group is eager to see God and to be a part of God’s work in the world.
  • By contrast, current field staff view internships as a ministry experience, something related to work and profession.  You can see Godin the word cloud, but on a much smaller scale, and almost no other spiritual terms.  Hopefully, this isn’t reflective of the spirituality of field missionaries!  Rather, it shows that they view internships through the lens of work activity.
  • Work dominates the current field missionary cloud.  They likely view internships not only as a lot of work for the intern, but also as a lot of work for resident missionaries.  Interns require a good deal of supervision and mentoring, which takes time.  There are immediate benefits in terms of enthusiastic service and future benefits in terms of long-term recruiting potential, but obtaining those rewards has a cost: work.
  • Both groups see experience as a key descriptor of mission internships.  But in what sense is the word used?  Is it about having a valuable experience (to be reflected upon) or is it about gaining experience (to be applied in the future)?  For current missionaries, the perspective is forward-looking, with future, preparation and potential appearing.  For prospects, there is a bit more balance.  While long-term and training are prominent, so is rewarding – suggesting intrinsic value, not just preparatory value.
  • Both groups see internships as an opportunity and are generally positive about them.  Good and great appear in moderate-to-large type in both clouds.  Current field workers also used the terms beneficial, excellent, positive and effective.
  • Both groups also are well aware that mission service of any length is challenging.
  • Time is a key element in internships.  In both clouds, the words time, term, long and short appear.
  • Financial considerations are an issue for prospective missionaries.  Support and fundraising aren’t the most prominent terms, but they are a real challenge/barrier for some prospects.  Those terms don’t show up on field missionaries’ radar as a key element of internships – they may assume that people considering long-term service will certainly be able to raise funding for a shorter-term opportunity (with those dollars easier to raise because they do not require a long-term commitment from donors).

What do you see in these clouds?  What are you concerned that you can’t see?