The pitfalls of self-assessment

 

This week, the eminently useful Brigada Today newsletter—in addition to drawing attention to GMI’s Agency Web Review—also posted an item from someone looking a self-assessment survey for local church missions programs, recalling that ACMC used to offer one.

 

Responses were subsequently posted by Marti Wade (who does great work with Missions Catalyst) noting that the tool is still available via Pioneers, which assumed ACMC assets upon its folding; and by David M of Propempo International, which also offers a version of the tool.  A snapshot of an excerpt from the ACMC/Pioneers version appears above.

Seeing the ACMC survey brought back a memory from a 2003 project that GMI did for ACMC.  We surveyed 189 ACMC member churches to understand the status of church mission programs as well as their needs and goals.  The survey included each of the twelve questions from the self-assessment grid.

Subsequently, we did statistical modeling to determine if/which/to what degree various church missions program elements were associated with growth in missions sending and with missions budget as a proportion of overall church budget.

Unfortunately, most of the correlations were not statistically significant, and those that were significant were negatively correlated—meaning churches that rated their mission program highly (placing greater priority on the various dimensions) tended to demonstrate less growth in sending or lower relative financial commitment.

How could this be?

Turns out that this is a fairly common outcome of self-assessment exercises.  In short, teams with excellent performance also tend to have high standards—and their vision for growth frequently leads them to be more self-critical than lower-performing teams, which often have lower standards.

So, am I discouraging local churches to use the Mission Assessment Tool?  Not at all.  I encourage churches to download it and use it as the basis for discussion—it can be a great discussion starter for vision building, clarifying core values and identifying priorities for development.  For the reason described above, you may find out that some team members differ on where the program stands—or where the opportunities are for growth.

But when program evaluation is the goal, it helps to have outside eyes providing the perspective.  Those well equipped to offer feedback on a church’s mission program are:

1. Those served by or in partnership with the mission team, such as missionaries who may have other supporting churches (these must be given anonymity in response) and/or

2. Outside consultants who work with many church mission programs and have a valid basis of comparison.

Meanwhile, at the 30,000-foot level, researchers, missiologists and consultants are eager to discover the key differences between high-performing church mission teams and others.  The statistical modeling sought to answer the question: What program elements are the most common outflows (or drivers) of increased financial/sending commitment: Better mission education?  Better worker training?  Greater emphasis on strategy?  More local mission involvement?  This is where self-assessment bias—seen across a sample of 189 churches—becomes a problem.

One helpful approach is to focus on relative data.  Were we to re-examine the analysis today, I would be inclined to transform the raw data into relative performance rankings (each church’s perception of its relative strengths and weaknesses).  This compensates for differing standards of excellence by looking at each church’s priorities.

Self-evaluation bias can also be reduced by developing assessment tools with response scales/categories that are so observably objective that they cannot easily be fudged.  The ACMC tool uses descriptions for each commitment level that are intended to be objectively observable—but in some cases they are subject to interpretation, or to cases where a higher-level condition may be met while a lower-level condition is unfulfilled.  In the 2003 study we gave specific instructions to respondents that they should proceed through each scale a step at a time, stopping at the lowest unmet condition.  However, such an instruction may not have been enough to overcome the need of some respondents to affirm their church’s mission program with high marks.

This issue also points to the importance of testing assessment tools for validity among a pilot sample of respondents—with results compared to an objective measure of excellence.

Take care with self-assessment.  After all, scripture itself warns us in Jeremiah 17:8-10 that “the heart is more deceitful than all else and is desperately sick; who can understand it?”

 

Using research to help create the future

This week the International Association of Missionary Aviation has its annual meeting in Idaho.  GMI board member Jon Lewis is the plenary speaker.

Five years ago, GMI played a significant role in IAMA’s annual meeting, as we presented the results of a multi-year research project looking at the present and future of mission aviation.  The idea behind the FlightPlan project was that global mission was trending away from an emphasis on overcoming physical barriers and moving toward an emphasis on overcoming political, cultural and religious barriers.  In such a world, what might be the appropriate – or potential – roles for the people and tools of the mission aviation community?

A cornerstone of GMI’s 184-page report was a set of seven prospective “models” for ministry.  These models emerged from an analysis of conditions and needs in mission and in general aviation – but also by looking at innovative enterprises in sectors that are “near neighbors” to mission aviation:

  • Organizations on the fringes of the mission aviation sector, such as Wings of Hope, a non-sectarian group that was later nominated for the Nobel Peace Prize.
  • Commercial entities that complement or parallel mission aviation, such as air taxi service and fractional jet ownership.
  • Organizations in the supply chain of mission aviation, such as aviator training schools and small-aircraft developers like Quest Aircraft.
  • Organizations that deal in similar activities to those of mission aviation, such as the global logistics industry and the UN’s World Food Programme.
  • Organizations that could be viewed as competitors to mission aviation, such as the Aga Khan Development Network, which sponsors community air service in the spiritual “Tension Belt” of Africa.

This near-neighbor approach is a systematic, intentional way of developing viable new models for business or ministry.  We didn’t think this up on our own; we borrowed the concept from Kim and Mauborgne, the authors of Blue Ocean Strategy (who are said to have built on ideas from Clayton Christensen and others).  They say:

The process of discovering and creating blue oceans [new models and markets] is not about predicting or preempting industry trends.  Nor is it a trial-and-error process of implementing wild new business ideas that happen to come across managers’ minds or intuition.  Rather, managers are engaged in a structured process of reordering market realities in a fundamentally new way.  (pp. 79-80)

That quote captures the kernel of using research in strategic planning.  Are you engaged in that process?

For the FlightPlan project, we used the process to identify seven models that organizations could use to challenge their strategic thinking and focus their strategic planning.  These included:

The Agile Provider: In a world where change is constant, this provider (or network) is ready for anything – broad and acute needs, short- and long-term deployment, people, skills and/or cargo. The Agile Provider has the resources, processes, flexibility and drive to deliver many resources in many places, on many scales, for many purposes – with the ultimate purpose of representing Christ to the world.

The Nation Developer: This knowledgeable organization assists in the development of transportation and communications infrastructure in nations that have a combination of spiritual needs and capacity-development needs, with focus on nations that have not been open to traditional forms of Christian witness.

The Field Opener: Because places remain where Christian workers – and thus the gospel – face physical barriers that prevent or delay access to God’s word and the Church, this provider (or network of providers) efficiently develops air access in remote areas, paving the way (sometimes literally) for others who are good spiritual and financial stewards of the access provided.

The Tribal Advocate: As tribal peoples direct the development and application of technologies to meet their current and future needs, knowledgeable Christian individuals and groups assist and advocate for them, honoring their decisions and partnering with them in carrying out their priorities and achieving their goals.

The Microaviator: A missionary, church or national church planter who uses one’s own plane as a personal vehicle to get from place to place quickly and safely, or who hires an air taxi service to do so. Microaviators typically use very small planes to do their work. They consider themselves missionaries first, aviators second. Microaviators may also include churches that charter business aircraft to transport short-term teams.

The Business Creator: An enterprise that uses business-as-mission strategies to establish aviation-related commerce, jobs and influence in cities and villages in less-reached areas. Independently, or in partnership with nationals, the Business Creator improves livelihoods for believers and unbelievers, builds goodwill, sets a positive example through faith and lifestyle, and creates natural evangelism opportunities.

The Resource Broker: This provider obtains, enhances and deploys valuable time and technology resources for aviation as they become available. The Resource Broker skillfully identifies, evaluates and capitalizes on resources that may be donated, loaned, salvaged, purchased at auction, etc. This low-cost, high-value approach enables the deployment of resources at an affordable cost for end users. Known for good stewardship and the ability to utilize resources that do not easily fit into traditional suppliers’ systems, the Resource Broker monitors aviation needs and opportunities to determine the best way to deploy resources.

Click here learn more about the FlightPlan research project and to download an executive summary of the research.

There are many ways to do futures planning and scenario research.  A good link to many resources for ministries is Jay Gary’s Christian Futures Network.

I know of at least two mission organizations that have done their own futures research with a high level of skill and intentionality.  One is Mission Aviation Fellowship, whose former COO David Bochman did such a project as part of his doctoral dissertation.  Another is Pioneers, though neither project has been published, to our knowledge.

What about you?  If your agency is interested in researching possible and preferable futures for your organization, let us know – GMI Research Services will be glad to help.