Chapter 40. Maintaining Quality Performance | Community Tool Box

• Why is quality important to you?

• What are the basic principles of TQM?

• What is TQM and its relevance to your organization?

• How do you achieve quality performance, using TQM and other principles?

• How do you maintain quality performance?

• The Fourteen Points

W. Edwards Deming, a businessman who was influential in American industry during World War II and Japanese industry afterwards developed successful management principles based on quality. These principles encouraged the development of a flexible, dynamic system which involved everyone in a company in the production of goods that exactly met the customer's needs, did precisely what they were supposed to do as effectively as possible every time at the best possible price, and were constantly being improved. His ideas are often referred to as Total Quality Management (TQM), and they have led to a number of similar theories of management and numerous innovations in businesses around the world.

Why is quality important for you?

Exactly what does quality mean in the context of advocacy, community development, health, or human service organizations or initiatives?

A quality program:

• Responds as effectively as possible to the needs it was designed to meet
• Is totally consistent with the mission and philosophy of the organization or group carrying it out
• Is sensitive to the needs and culture of the target population
• Is a model of ethical behavior

But why is quality important for a grass roots organization?

• Quality makes a group more effective at meeting the needs it's concerned with
• Quality adds strength and credibility to your organization or initiative
• Ethically, you're bound to provide the absolute best quality of service or advocacy you can
• Quality is always more economical in the long run

Developing a "culture of quality" can have a number of positive effects on your organization itself

• If staff members and volunteers know that they and the organization are doing the best job possible, it builds their morale and makes them proud of themselves and the organization
• Striving for quality helps to develop organizational and individual competence, thus continually improving the organization
• A quality program continually increases its performance level and improves its service delivery, which gives your organization credibility and ultimately benefits your target audience

What are the basic principles of TQM?

(Much of the following discussion is based on material contained in Introduction to Total Quality: Quality Management for Production, Processing, and Services, 2nd Edition. Full source citation can be found under Resources.)

There are some basic assumptions that underlie the idea of TQM. In this section, we'll look at how they might relate to your organization or initiative.

Key elements of total quality

• Customer Focus: Everything an organization does should have the needs of the customer as its starting point. In your work, the "customer" is the target population or the community that will benefit from what you are offering or doing. What are the needs to which you are responding? How can you meet those needs effectively, appropriately, and with respect for the people you're intending to serve?
• Obsession with Quality: Quality has to be something that's considered from the very beginning and built into everything a business or organization does. Planning carefully, monitoring your work, and constant reevaluation and adjustment are all extremely important. You don't ensure quality by catching mistakes before they reach the customer; you ensure it by setting up a system in which you don't make the mistakes to begin with. Everyone in the organization must understand and adopt this point of view if the organization is truly going to have quality performance.
• Continual Improvement of Systems: The work of an organization must be viewed as a process that is never finished. Any program can always be improved, and must be changed as the needs of the community or the target population change.
• Unity of Purpose: In order for quality to be achieved, everyone in an organization or business has to work together toward common goals. That means mutual support throughout the organization, not turf battles, not jealousy, not unnecessary competition. All interactions among people in the organization should be mutually helpful and aimed at achieving the best possible performance of the organization as a whole.
• Teamwork: Working in teams, rather than individually, people make better connections with their colleagues and the organization, and create better results. Teamwork removes performance pressure from the individual and usually coaxes better performance from everyone.
• Employee Involvement: If everyone in an organization is to be committed to quality performance, then all staff members should have the ability to contribute to its achievement. That means that people must have enough control over their own jobs to do them effectively, and that everyone's opinions and ideas must be respected and taken seriously.
• Education and Training: Achieving quality requires constant learning for everyone in an organization, and that learning needs to be part of the organizational culture. Not only should staff members be learning from others in the organization, but they should also be encouraged to take courses, to attend organization-sponsored trainings and workshops, to visit other organizations, etc., to continually learn more about their work, and to get new ideas and perspectives on it.
• Scientific Approach: For grass roots and community-based organizations, this means using the best research available, as well as the experience of others, to construct an effective program or initiative. That approach is much more likely to result in success and high quality than relying only on intuition or on what seems politically correct.

The founders of a Massachusetts community-based adult literacy program, with backgrounds in both developmental psychology and reading theory, based their program on the best available research in both areas. They made sure that the educational and support elements of the program fit together properly, and trained staff with that in mind. Initially, since it was doing something that hadn't been done with adults before, the program was severely criticized by others in the field. The founders were accused of cheating their students by not using a strict, phonics-based approach to reading, and by paying too much attention to other matters - students' concerns, community issues, etc. As time went on, however, and the program's drop-out rate remained extremely low and its students' success rate extremely high, others began experimenting with similar ideas. 15 years later, the program is a model for the state, but it keeps changing, responding both to student needs and feedback and to new research findings.

• Long-Term Commitment: The best work in the world is ultimately useless if it's not maintained. Quality is a long-term concept: you have to keep striving for its improvement, even after you've achieved an acceptable level of performance. "Acceptable" is never good enough. In fact, you're never really at an endpoint, because the level you're trying to reach is "the best that can possibly be."

The Deming Cycle

The assumptions above underlie the "Deming Cycle," which is really a process for creating and selling a quality product. We'll revisit the Deming Cycle later to examine how it can be used in an advocacy, community development, health, or human service context.

• Plan - conduct consumer research and use it for planning the product
• Do -  produce the product
• Check - check the product to make sure it was produced in accordance with the plan
• Act - market the product
• Analyze - analyze how the product is received in the marketplace in terms of quality, cost, and other data

What is TQM and its relevance to your organization?

It is important to note that the principles of TQM were designed for the business sector, so while some aspects are relevant to organizations concerned with advocacy, community development, health, and human services, other aspects may be contrary to the goals of those organizations.

Some elements of TQM that would work toward quality in any environment include:

• The need for careful planning, monitoring, evaluation, and adjustment
• Teamwork and the empowerment of all in the organization
• Constant education and training for all staff
• Attention to the needs of the target population and to the results for them
• Identifying and changing what doesn't work well
• Encouraging and rewarding, rather than discouraging, new ideas
• Developing an organization-wide culture of quality
• Keeping at it over the long term

Some elements which may not work toward quality for your organization are those that assume that the goal is the success of a business, such as:

• The emphasis on products and production, which may take the focus off the human needs and consequences your organization is concerned with
• The assumption of a hierarchical structure where those in authority "let " others have a say in the achievement of quality, and where leadership always comes from the top, which may conflict with the way your organization operates
• The definition of everyone as either a supplier or a consumer/customer, which may provide the wrong metaphors for grass roots work where everyone is, on some level, a participant

How do you achieve quality performance, using TQM and other principles?

Using the Deming Cycle while keeping some of the basic TQM principles in mind can help you design, deliver, refine, and maintain an effective program or initiative.

Plan

Conduct consumer research and use it for planning the product. The "product" here is the actual program you intend to conduct, and the "consumer research" is an examination of actual needs of the target population, the community, and others who will be affected.

Thus, the "Plan" part of the cycle might include the following:

• Conducting a needs assessment, involving everyone concerned
• Deciding what the desirable outcomes are, from the perspectives of the target population, the organization, and the larger community
• Determining ways to reach those outcomes that are feasible, consistent with the guiding principles of the organization, inclusive (respectful of all and beneficial to as many people and groups as possible), and consistent with the needs and culture of the target population
• Developing indicators to show when you have reached either outcomes themselves or significant points on the way to reaching those outcomes
• Inviting all stakeholders to participate in the development of the plan

Do

Produce the product. The "production" part of the process is the actual design of the program, outreach effort, treatment strategy, etc. that will meet the need determined in the "Plan" part of the cycle. Much of the actual work here depends not only on TQM principles (teamwork, employee involvement, scientific approach, obsession with quality, and customer focus), but also on common sense and organizing principles.

The following are important elements of designing an effective program:

• Finding out what has already been tried in the community, and how well it worked
• Discovering whether there's any residual bad feeling attached to certain methods or approaches -- or people -- which may resurface if they're proposed again
• Using as examples other communities that have successfully mounted similar programs, while remaining aware that not everything that works in one place will work in another
• Consulting the research to see what has worked in this situation
• Involving all stakeholders in the development of the program or initiative, especially the people who will do the actual work
• Taking care of the logistics: a place to operate, equipment and supplies, the proper staff and/or volunteers on board, etc.

Check

Check the product to make sure it was produced in accordance with the plan. Compare the details and overall shape of the program or initiative to the plan. Does it align with the needs assessment? Does it look like it will address the desired outcomes in desired ways? Is it inclusive? Was everyone involved in its development? Is it feasible? Is it ready to go?

Act

Market the product. "Marketing the product" here means actually running the program or initiative that you've planned.

If it's going to work well, there are some non-TQM standards that need to be applied:

• Everyone involved should understand the process that led up to this program, as well as the philosophy, concept, and workings of it
• Everyone involved should be committed to making every effort to bring about success. A program or initiative should never fail because people don't follow through or do their jobs. (This doesn't mean that you shouldn't expect mistakes; it means, rather, that mistakes shouldn't happen just because people weren't trying, or because they simply didn't bother to do something they knew they had to do.)
• All the planning in the world is useless if everyone involved doesn't go into the experience expecting to do their best, and if there aren't good people implementing the functions of the organization

Analyze

Analyze how the product is received in terms of quality, cost, and other data. Analysis in this context - looking at what you're doing, evaluating it, and trying to improve it - needs to be conducted on the basis of the original plan, with discussions among participants, staff, and others.

Consider:

• Does the program or initiative actually address the identified needs? Are these needs the same as when the original assessment was conducted?
• Does the program or initiative reach, or help participants reach, the desired outcomes? Were those outcomes the right ones to aim for, or do they need to be changed? (Looking at the indicators you've developed should help you answer both these questions.)
• Is the plan in fact feasible? Can the program or initiative be run with the time, resources, and personnel available? Is it accessible to participants? Are staff and volunteers able to do their jobs without having to work to exhaustion, or beyond reasonable expectations? Is the program or initiative accepted by the community and other organizations?
• Is the program or initiative consistent with the vision, mission, philosophy and guiding principles of the organization (and are those still the same as when the plan was formulated)? Does what actually goes on in the program or initiative -- working conditions, empowerment, relations among staff, participants, volunteers, and the community -- mirror its desired effect on the community and society?
• Is the program or initiative inclusive and respectful of the target population and the community?
• Is the program or initiative ethical? Are you skirting your own principles in any way? Are you ignoring basic principles of fairness, honesty, civility, democracy, or responsibility for your actions?

Although the two are usually congruent, ethical is not always exactly the same as legal. The exercise of civil disobedience may be profoundly ethical, while being, at the same time, inarguably illegal. The civil rights marches and actions led by Martin Luther King and others during the 1950's and 60's often fell into this category.

Restart the cycle

Your analysis should naturally lead into a new planning sessions, and where necessary, lead to rethinking and reworking the program or initiative, or even the task of the organization itself.

How do you maintain quality performance?

While the maintenance of quality is, to some extent, built into the Deming Cycle, it requires some particular commitments and action.

Institutionalization of dynamism

An organization needs to be dynamic, always moving and always seeking continued improvement, and to institutionalize its dynamic character.

This means:

• An assumption of dynamism needs to be part of the organizational culture, with everyone understanding and buying into it.
• Encouraging and providing support -- to staff, volunteers, and participants -- for learning.

Organizational support for education includes providing, or coordinating with another organization's provision of, professional development or university courses, training, certification, etc., that help staff and volunteers to build their skills. If financial resources are not available, other means of institutional support -- release time or leave time, special recognition, a library for the organization, study circles or reading groups, etc. -- need to be considered. Everyone in the organization, including administrators and Board members, should be encouraged to take advantage of learning opportunities and to model learning behavior.

• Listening to and carefully evaluating ideas from everyone.
• Encouraging openness to change and experimentation with new ideas and strategies

Administrators and Board members need to model such openness by being willing to reexamine and change procedures, policies, etc. when needed. Staff and volunteers should be given room to try out even things that others may be skeptical about, as long as they can justify the attempt ("It worked elsewhere" and "I learned about it in a course" are both reasonable justifications.). If such an attempt is honestly carried out, it should be seen as a positive even if it fails: it provided new information, and is another building block in the construction of a quality program.

• Never being complacent and always being open to the idea that the work could be done differently -- and better.
• Incorporating constant reevaluation, including feedback and ideas from the target population.
• Always being aware of the original mission, but not being afraid of change. While the mission itself may change as the community and circumstances do, it should nonetheless remain consistent with the principles and philosophy upon which the organization was founded.

Long-range strategic planning

To maintain quality, an organization needs to continually look at itself over and for the long term.

It needs to ask some questions about its role and its future:

• Is it meeting an ongoing need effectively? If not, what does it need to do to become more effective?
• Have community needs changed? Are they likely to? If so, how can the organization regroup to meet new needs?
• Are there more or different things it should be doing? Does it need to expand its present activities to meet current or projected community needs?
• Does it need more resources, or will it in the future? What are some likely sources?
• Is its structure appropriate to what it's doing, and consistent with its mission and guiding principles? (An organization dedicated to empowerment, for instance, may not be consistent if its internal structure is hierarchical and authoritarian.)
• Are its goals, vision, and philosophy still relevant to the realities of the community and in keeping with its organizational mission and guiding principles?

SWOT Analysis

One way to explore these questions is through the use of another device partially borrowed from TQM: SWOT analysis. SWOT stands for Strengths, Weaknesses, Opportunities, and Threats. Each of the questions above can be examined in the light of SWOT analysis. What are the strengths and weaknesses of your organization in regard to each question? What opportunities exist for the organization in each area of its functioning? And what threats or challenges will the organization have to overcome if it is to continue to be successful, and to maintain quality performance?

Strengths and weaknesses: Strengths and weaknesses may be trends, rather than specifics. A level of service that is currently appropriate, for example, is not a strength if it is more or less than will be needed in a year. A new program that's not ready to implement yet is not a weakness if it's unready because the developers are taking the time necessary to make it effective.

By the same token, strengths and weaknesses don't necessarily lie only in the success of programs or the skills of staff members, but in such areas as relationships, contacts, and reputation. An organization running a great program may still be have serious weaknesses because it lacks some of these other features, no matter how well it carries out its day-to-day activities.

Opportunities: Opportunities can take many forms.

• An organization may be able to meet other needs with its current structure. For instance, an organization that publicizes and provides prenatal care to pregnant teens could be in an excellent position to also publicize and provide vaccinations, nutrition information, and help with parenting skills after the babies are born.
• It may be possible to expand into other areas of service, or into a larger arena (another town, another county, national instead of just one state).
• Increased funding may be available from new sources, or because of changed circumstances. A new census, for example, can result in an increase in federal funds to a region, or an economic downturn may bring a demand -- and increased funding -- for adult education or retraining.
• Collaboration with other groups, leading to increased resources, may become a prospect.
• Invitations or awards offered to your organization or staff members or good press may lead to your organization being viewed as more "legitimate."

Taking advantage of any opportunity can have both positive and negative consequences for your organization, so it's important to analyze the situation carefully before committing yourself.

Threats (Challenges): Some of the challenges that go along with any opportunity can be truly daunting if they're not thought through carefully. Many of the opportunities above require some sort of organizational restructuring or growth, processes that are always difficult, and require a lot of planning. Some even represent rethinking the purpose of the organization, which may become a different organization in the process. In becoming larger or more accepted, for instance, an organization may forget its roots or its guiding principles, and lose much of its effectiveness.

Other threats may come unaccompanied by opportunity. Your organization may experience difficulty finding -- and keeping -- ongoing funding and other resources, including competent staff; sustaining continued effort in all areas of functioning (advertising, recruitment, public relations, programming, evaluation, etc.); dealing with controversy; and addressing antagonism from individuals, other groups, or the community.

Applying SWOT analysis to all the areas your organization has to deal with makes it easier both to anticipate and prepare for the negative, and to remember to identify and build on the positive.

Other facets of the planning process

Some specific areas that long-range strategic planning needs to address at regular intervals:

• Reexamining the organization's vision, guiding principles, and mission statement. Are they still relevant to what the organization does, and are they still what the organization believes? Do they need to be restated or redefined?
• Reexamining the goals of the organization. Are they still relevant to the needs they were originally meant to address? Are they consistent with the vision and mission statement?
• Reexamining the current strategies of the organization for meeting those goals. Are current methods effective? Are they consistent with the organization's vision, mission statement, and guiding principles? Are they feasible? Do they play well in the community? Are they inclusive and respectful? Are they ethical?

Keeping at it

The single most important thing to understand about maintaining quality performance -- or maintaining an organization, for that matter -- is that you can never stop working at it. No effort at maintaining quality will work any longer than it is applied. No matter how institutionalized dynamism becomes, no matter how good your planning process is, they take constant care.

The Fourteen Points

Deming used the key elements listed above in "Basic principles of TQM" to define the "Fourteen Points," fourteen things that he felt businesses needed to do in order to achieve quality performance.

1. Create constancy of purpose toward the improvement of products and services in order to become competitive, stay in business, and provide jobs.
2. Adopt the new philosophy. Management must learn that it is a new economic age and awaken to the challenge, learn their responsibilities, and take on leadership for change.
3. Stop depending on inspection to achieve quality. Build in quality from the start.
4. Stop awarding contracts on the basis of low bids.
5. Continuously improve the system of production and services to enhance quality and productivity, and thus constantly to reduce costs.
6. Institute training on the job.
7. Institute leadership. The purpose of leadership should be to help people and technology work better.
8. Drive out fear so that everyone may work effectively.
9. Break down barriers between departments so that people can work as a team.
10. Eliminate slogans, exhortations, and targets for the work force. They create adversarial relationships.
11. Eliminate quotas and management by objectives. Substitute leadership.
12. Remove barriers that rob employees of their pride of workmanship.
13. Institute a vigorous program of education and self improvement.
14. Make the transformation everyone's job and put everyone to work on it.

In Summary

Achieving and maintaining quality performance is important to the target population, to funders, and to the community.

Using some TQM principles and, specifically, the Deming Cycle (Plan, Do, Check, Act, Analyze) can be helpful in getting to a high level of quality and continuing to improve.

In general, achieving and maintaining quality is a result of:

• Careful planning
• Program or initiative development that adheres to that planning
• Implementation that takes quality into account
• Constant reevaluation of implementation and of the organization
• An assumption of the dynamic character of the organization, and a willingness to change continually in striving for a better way to accomplish goals
• Keeping at it indefinitely

If you can carry out and institutionalize these steps, especially the last, success is in your grasp.

What is the Total Quality Management system?

___Total Quality Management refers to principles developed by W. Edwards Deming that encourage the development of a flexible, dynamic system which involves everyone in an organization in the production of goods that exactly meet the customer's needs, do precisely what they are supposed to do as effectively as possible every time at the best possible price, and are constantly being improved

Why is quality important for a grass roots organization?

___It increase the group's effectiveness at reaching its goals

___It adds strength and credibility to your organization/initiative

___It helps the organization fulfill the ethical obligation to provide the best service or advocacy possible

___It is more economical over the long run

___It has many positive effects on your organization

What are the basic principles of TQM?

___Customer focus

___Obsession with quality

___Continual improvement of systems

___Unity of purpose

___Teamwork

___Employee involvement

___Education and training

___Scientific approach

___Long-term commitment

What are the steps of the Deming Cycle?

___Plan: Conduct research and plan for improvement

___Do: Produce the product

___Check: Check to be sure that the product meets planning goals

___Act: Market the product

___Analyze: Analyze how the product is received in the marketplace

How should you use TQM for your organization?

___Determine which elements of TQM are relevant to your situation, and which are not

___Commit your organization to maintaining quality performance through:

• Institutionalization of dynamism
• Long-range strategic planning
• Keeping at it forever

___Know what SWOT analysis is, and how to use it

A new community health initiative was about to start, funded by anti-smoking money from the state Department of Health. The program was aimed at people in a number of community service programs – employment training and job development, adult basic education, teen pregnancy prevention, etc. Its focus was supposed to be on teaching people why smoking was unhealthy and addictive, and helping them to quit.

Before the program started, however, its organizers decided that they needed to know what health issues potential participants were concerned with.

The health educators interviewed a large number of people involved in the community programs they were targeting, and found that a majority of them both smoked and were interested in quitting. Far more important to these folks, however, were the related problems of neighborhood violence and substance use.  Smoking was way down the list for them.

After conferring with its funder, the community health program, as a result, began by concentrating on violence and substance use as community health issues. It attracted a large number of participants, who were able, through a variety of strategies, to reduce violence in the area and to bring about the establishment of a substance use treatment center in the neighborhood. In the course of their work on these issues, participants also were introduced to information on tobacco addiction, and to methods for smoking prevention and cessation. As a result, a great majority of the smokers in the program joined smoking cessation groups and were able to quit. Although they had started out focusing on what seemed to be unrelated health issues, the health educators found themselves with one of the most successful smoking cessation programs in the state. They realized that it never would have happened if they hadn’t asked participants for feedback about what they needed, and what would be likely to draw them in.

Health and community service programs almost always have the best interests of the community at heart, but they don’t always know what those best interests are or the best way to approach them. One of the easiest ways to find out is to ask the people who know best – those at whom a program or intervention is aimed. Chapter 40 is about attaining and maintaining quality in your work. In this section, we’ll discuss what feedback from participants consists of, how you can get it, and how you can use it to create and maintain a high-quality program, initiative, or intervention that meets the real needs of the community.

What do we mean by feedback from participants?

In simplest terms, feedback from participants consists of reactions to or opinions about your effort from those who are affected by it. If you’ve read the first section of this chapter, you’ll remember that an important element of building quality into programs is customer-centeredness. That means doing what the customers actually need and want, rather than what you think they want. The customers, in this case, are the folks you’re hoping will benefit from your work. To be customer-centered, you have to involve them in planning and evaluating what you do at all stages, so that you can adjust it to their needs as much as possible.

The idea of customer-centeredness fits nicely with the emphasis in the Community Tool Box on participatory process. We believe that involving participants from the beginning in a program or intervention greatly increases the possibility of its success, largely for the same reasons that W. Edwards Deming, the father of Total Quality Management, deemed that it was important to consult the customer.  Knowing what people actually need makes it much easier to provide it.

Feedback is not simply criticism, constructive or otherwise. Rather, it is a reflection of the perceptions of the person or group providing it. You may think you’re presenting things in a particular way, or accomplishing a particular purpose by what you’re doing; but if those at whom it’s aimed see it differently, you may have little chance of success, or of achieving what you set out to do.

Obviously, feedback can be positive, negative, or neutral.

Its nature may also be influenced by some other factors:

• It can be completely or at least somewhat objective, based on facts, observation, and past knowledge; or almost entirely subjective, based on emotional reactions, defensiveness, loyalty (to family, friends, or class), cultural gaps, or distrust of outsiders.
• It might be bolstered by participants’ knowledge of the history of the community, of specific people and relationships, of culture, or of other factors and information.
• It might be limited by participants’ limited knowledge of an issue or process, by their lack of information, or by their lack of understanding of the information available.
• It might be solicited (asked for) or unsolicited. Feedback that’s volunteered often takes a negative – sometimes an extremely negative – form, and may be hard to pay attention to and incorporate into your work.

In addition to its range of possible characteristics, feedback can be delivered in a number of ways, and it’s important to pay attention to as many of them as possible.

• Direct verbal feedback. The most useful feedback, in most cases, is direct and to the point. That’s the kind you get when you ask someone what they think, and they’re willing to give you an honest answer, either face to face or in writing.  Unfortunately, that kind of feedback isn’t always available.

In general, this section deals with obtaining intentional, direct feedback.  We include some other options immediately below, and it’s important to take them into account, but it’s a lot easier to interpret and use direct feedback if you can get it.

• Indirect verbal feedback.  This might include remarks that aren’t meant specifically as feedback, parts of conversations overheard in passing, or comments that may not, at first hearing, seem much like useful feedback at all.

A conversation with a participant might include a remark about how great the program has been for her, or how a particular activity or process simply wasn’t helpful. Participants may say things to one another that they’d be reluctant to pass on to a staff member, and occasionally those might be overheard, or reported by the hearer.

Comments that initially do not seem useful may just need some additional interpretation. A common remark, for instance, especially among young participants, is “This is stupid!”  While it sounds like a generalized criticism based on a cynical view of the program or the world, or on a lack of understanding of the situation, it often has a very specific meaning: “This makes me feel stupid.”  Learning that what you’re doing has that effect may be a very important piece of feedback, and can lead to useful changes.  Instead of ignoring the comment, or making light of it, you might ask “Why is it stupid?” or “What would make it less stupid?”   The answer might be tremendously useful to both of you.

• Direct non-verbal feedback.  When participants fail to keep appointments, never follow through on tasks, or simply disappear from the program, they’re often giving you feedback.  Sometimes – with children or youth, very rarely with adults – this non-verbal feedback may even encompass acting out and outright violence directed toward staff members or other participants.  It’s extreme, and like other non-verbal feedback, doesn’t always have to do with the nature of the program or relationships with the specific people it’s aimed at.

It’s important to draw the distinction between feedback – actions or comments that indicate a real reaction to a real situation or aspect of a program – and symptoms of a larger individual issue. A teen may be hostile or withdrawn because what you’re doing seems insulting or demeaning to him, or because his history of abuse and neglect is so powerful that it might be years – or never – before anything can reach him.  A number of people missing appointments or leaving the program may be a reaction to the way they’re being treated or to the ineffectiveness of the methods you’re using; but it may also be a reflection of other pressures, illness, or domestic situations.

• Indirect non-verbal feedback.  People send all kinds of messages with their facial expressions, tone of voice, body language (posture, hand or body movements, eye contact, etc.), and their actions.  Participants are often extremely nervous or scared – or defiant – when they start, but that usually breaks down as they become more comfortable with the situation and the other people – staff members and other participants – involved in it.  If it doesn’t change – if the fluttering hands, shaky voice, and other signs of nervousness don’t ease for most participants, or if they remain hostile or withdrawn – they’re telling you something about their response to what you’re doing.  If they never follow through on tasks or assignments, continually fail to keep appointments, or simply disappear from the program, they’re making a feedback statement that you’d better listen to.

It’s important to remember that feedback is a matter of perception.  Thus, even though it comes from participants, that doesn’t mean it’s accurate.  It must be weighed as any other feedback would.  What is true, however, is that if many participants perceive that they’re not being well served, that perception needs to be changed whether it’s correct or not.  Even if feedback is offered in a negative way, you should pay attention.

Why do you need feedback from participants?

• It’s part of being customer-centered. Paying attention to what participants need and want is a crucial element in creating a quality program.
• It gives you tools to improve your program. Understanding what’s needed and what isn’t, and what’s working and what isn’t helps you to increase the effectiveness and success of your work.
• It allows you to respond to changes in the community, the population, or the situation. There may be social or cultural changes within the population you’re working with (an increase in immigration, for instance), changes in the community (a downturn in a major industry), or other factors (an increase or decrease in neighborhood violence) that affect your program.  Being able to spot and react to these through participant feedback will keep your program current and effective.
• It can give you information about the history of the community, the history of your issue in the community, or the history of your population. Hearing from participants that what you’re doing has bad connotations in the community, or that it’s been tried before, and failed as a result of conditions that haven’t changed, can help you create a program without the baggage of past mistakes.
• It can inform you about personalities and relationships in the community and/or the population you’re working with. This kind of knowledge can be very important when you’re trying to get different sectors of the community or different neighborhoods working together, or when you’re trying to put together a planning or other group.
• It can tell you when your methods or approach aren’t working. If participants aren’t feeling like there’s any progress, they may be right, unless you have evidence to the contrary.
• It can tell you when you’re treating people in ways that make them feel uncomfortable, angry, or otherwise disrespected  Honest participant feedback will let you know if your intake procedure is too impersonal and scares potential participants away, or if your attitude toward the teens in your program is that they don’t know anything and are nothing but trouble.
• It can help you deal with what’s important to participants, even though that may not be your major goal. The story of the smoking cessation program that opens this section is an example of how paying attention to what participants are concerned about can pay big dividends in the long run.
• It can assure that your objectives make as much sense for participants as they do for you. Regardless of your long-term goals, you have to start where participants are.  If they see the program as irrelevant to them, all you’ll see are their backs.

Who are the participants that can offer feedback?

When we talk about conscious, intentional, reasonably honest feedback, the typical providers are adults and teens with no intellectual, psychological, or social barriers to providing it.  These are the folks who are most likely to be able and willing to take part in interviews, fill out questionnaires or surveys, and be involved in the assessment, planning, implementation, or evaluation of programs in which they participate. That doesn’t mean, however, that these are the only people you can ask for feedback, or from whom you can get it. To a certain extent, it depends on what kind of feedback you’re looking for, and how hard you’re willing to look for it.

Other possible providers of feedback include:

• Speakers of languages other than the majority language. Obviously, if you can speak their language, or if you have access to an interpreter, these participants can provide the same kind of feedback any other adults and adolescents can.  You have to understand their culture, however, to understand how direct they’ll be with you.  In some cultures, it’s unthinkable to criticize anyone in authority; in some, it’s rude to criticize directly at all.  Others may be afraid to say anything negative, for fear of reprisals; if they’re undocumented, they may be afraid to say anything.
• People who have left the program without completing goals. Their comments may be the most important.  Why did they leave?  What could have kept them in the program?  What would attract them back?  The effort to track down and follow up with these short-time participants can be rewarding.

In these cases, it’s often not what the program did, but what it didn’t do that proves important.  Participants may have needed child care or transportation, which weren’t available.  They may have needed an extra bit of attention that they felt they weren’t getting.  Understanding these kinds of needs won’t solve everyone’s problem or keep every participant in the program, but it can certainly help many.

• Children. Children may be limited in their comprehension of how a particular program or service affects them, but they certainly know how they feel about it, and how they respond to it.  An advantage of asking children for feedback is that they are usually willing to give it.  If  you ask the right questions, you can get a lot of useful material from them.  Children are also more likely to give other kinds of direct feedback – acting out, for example.  That’s a lot harder to deal with, but it can convey important information, nonetheless.
• At-risk or troubled adolescents. Keep in mind that there are many factors at work here.  All teens, but especially those at risk because of violence, school failure, substance use, or other problems – may have to deal with a self-image and a real or imagined reputation among their peers that dictates never saying anything positive about the adult world. You have to win their trust – often a long process – before they’ll open up and tell you what they’re really thinking.
• People with mental illness. They can often provide good information on the services they’re receiving, or on other programs they’re engaged in, and have a great deal of relevant information to offer if you’re willing to listen and sort it out.
• Developmentally delayed teens and adults. The level of feedback you’re likely to get from these folks depends to a large extent on their level of functioning. They can certainly tell or show you whether they’re happy or unhappy to be there, but they may or may not be able to explain why.

When might you obtain and use feedback from participants?

In thinking about when feedback is necessary, it’s helpful to refer to the Deming Cycle, which consists of five phases: Plan, Do, Check, Act, Analyze. Feedback from participants can and should be used at every phase of the Deming Cycle.

• Before a program or initiative starts (Plan): Participant feedback is crucial to ensuring accurate assessment of the assets and needs of the community, and to understanding what the appropriate outcomes of an effort should be.
• During the development of the program or initiative itself (Do): Having members of the population you’re aiming at provide feedback for – or, better yet, engage in – the planning and structuring process will greatly increase the chances that your effort will attract participants and meet their needs.
• When your program is ready to begin (Check): Asking potential participants for feedback on your plan can identify bad choices, omissions, and strong points, and help you to make your effort as effective as possible before you begin. Be sure to ask if it addresses the needs expressed by potential participants, and if it is likely to achieve the outcomes that meet those needs.
• During the running of the program (Act): Continuing to ask for and obtain feedback from participants as the program runs will allow you to make adjustments on the fly, and to fine-tune the program as it runs.
• As part of a formal or informal evaluation (Analyze): You should conduct some sort of evaluation on a regular basis (annually is typical). You really can’t get an accurate picture of the strengths and weaknesses and the effectiveness of your efforts without participant feedback. The evaluation allows you to incorporate that feedback, and other information into an improved effort for the following period, thus starting the cycle over again.

How do you obtain feedback from participants?

Obtaining honest feedback from participants should be simple – just ask them, right? Sometimes that’s all you have to do (although you do have to do it – you’re not likely to get much feedback without asking), and you’ll get all the feedback you want, and more. But in many cases, it’s not true at all.  We’ve mentioned the adolescents who can’t, for reasons of self-image and reputation, bring themselves to say anything good – or sometimes anything at all – about adult-connected activities.  And there are cultural differences – for example, the folks who won’t ever say anything negative about or to anyone they see as an authority figure or a high-status person (a teacher, doctor, or administrator, for example), or to someone above a certain age.

Other cultural issues can arise around gender roles (men who are unused to dealing with independent women, for instance, or women who are fearful of offending men), mistrust of other cultures, or simply a lack of understanding about what kind of information would be useful. Many people, culture aside, are eager to please, and will try to say what they think you want to hear. Language and literacy difficulties can present their own problems, as can distrust of or hostility toward any kind of authority, whether actual or perceived (an especially difficult issue in programs that aren’t voluntary, such as court-mandated substance use treatment). Some participants may fear that they’ll be thrown out of the program if they criticize it; others may simply not want to offend someone they like as a person.

It’s our assumption that obtaining honest feedback is easier if participants have felt, from the beginning, that they have a stake in the program.  If they’ve been invited to participate in planning and implementing their own service or education or treatment, they’ll be far more comfortable providing honest and useful feedback on the process, content, and results of it.  If they see themselves, and are treated, as partners in the enterprise, working with staff toward a common goal, providing feedback will be natural.  If they see themselves, and are treated, as clients who have no important skills or knowledge, it will be like pulling teeth.

We present below several ways to ask for and get good feedback from participants, but we believe that the easiest way to assure it is to invite participants to be full partners from the outset, where possible.

Methods of obtaining feedback

There are a number of methods you might use to obtain feedback.  Which is best will depend on your situation, and you may in fact use more than one, in order to reach different groups of participants.

Surveys. Using a survey to collect feedback is a common practice.  Surveys are usually administered in one of four ways.

• Written. This often takes the form of a short, multiple-choice questionnaire that people can fill out and send or hand back.  Some surveys are much longer, however, and may ask for comments as well as yes-no answers or ratings on a scale.
• Face to face.  Here, an interviewer asks specific questions – also usually short and often multiple choice – and records the answers.  If the survey is much longer, it essentially becomes an interview, a method we’ll discuss in a little while.
• Phone.  This is similar to a face-to-face survey, but, because the two people involved can’t see each other, it is somewhat more impersonal.
• Online surveys. These are now quite common. However, keep in mind that if you’re working with a low-income population, this isn’t the best method, simply because a large number of the people you want to reach are unlikely to have access to a computer.

E-mail and Internet surveys are becoming more common, and may be something you can use. In general, if you’re working with a low-income population, this isn’t the best method, simply because a large number of the people you want to reach are unlikely to have access to a computer.

Like all the methods described here, surveys have their advantages and disadvantages. Among the advantages are that short surveys are relatively easy to complete, so that people are more likely to actually fill them out.  They are limited to specific questions, and generally don’t ask for too much more than checking a box or circling a number for each one. Written surveys can also easily be administered so that the people filling them out remain anonymous, and thus take no risks.

Getting people to fill out and return surveys can be difficult.  Offering incentives – entering anyone who returns the survey in a prize drawing, for instance – can help, as can presenting the survey so that it has to be filled out and returned on the spot.

Among the disadvantages are the limited amount of information they often provide. Long surveys can be very informative, but they’re harder to complete, and fewer of them are returned. Long phone surveys can help with this issue, but they run into time – which often translates to money, if you’re paying the phone interviewer, or paying phone charges by the minute. People may be less willing to cooperate, both because of the time involved, and because, promises of anonymity aside, the interviewer knows who they are.

Furthermore, participants may have difficulty reading, as is the case for many high-school dropouts and for people in such programs as adult basic education and English as a Second or Other Language (ESOL). In this case, either the surveys have to be administered face to face or by phone, or someone has to read the survey to a group or individual to make sure that they understand each question.

Interviews. These are face-to-face conversations with a specific purpose – in this case to get feedback about your work.  Interviews can be conducted one-on-one or in a group;  we’ll look at individual interviews here, and discuss group interviews below.

An interview is different from a face-to-face survey in a number of ways. First, although interviews are sometimes standardized (everyone is asked the same questions in the same words), it is also common for the interviewer to have some leeway.  While there may be a specific list of questions that the interviewer wants the answers to, they might not be phrased the same way for every person, the way survey questions are.  The interviewer might be free to ask other questions as well, or to comment on what the interviewee says. Furthermore, interviews usually aren’t designed to be as brief and easy as possible – they may go on for an hour or more.

Most important, perhaps, is that the answers the interviewer seeks are not yes/no or multiple choice.  Just the opposite, in fact: most interviewers try to ask open-ended questions, questions that can’t be answered with a simple yes or no. (Rather than asking “Did you like X?”, for example, an interviewer might ask “What did you like or dislike most about X?  Why?”)  A good interviewer will also ask follow-up questions when she hears what seems like an important or interesting comment. (It sounds like the other people that you met in the program were really important to you.  What do you think it was about that group that had such an impact?)

Individual interviews are great vehicles for collecting large amounts of detailed information. They offer the interviewee privacy and the interviewer’s undivided attention, both of which act as invitations to be open and thoughtful, and they give the interviewer the opportunity to really concentrate on one person’s thoughts.

On the minus side, individual interviews are time-consuming, and as a result may limit the number of people who can be consulted. Furthermore, in an individual interview, there’s generally no check on the accuracy of an interviewee’s stories or allegations – no confirmation or denial from another participant.  Finally, for an interview to be really successful, the interviewer and interviewee have to establish a personal connection. When this doesn’t happen, the interview may be less useful.

Small-group interviews and focus groups. A small-group interview, as the name implies, is similar to an individual interview, except that it includes several people instead of one.  Usually limited to perhaps six to eight participants, small-group interviews follow the same informal “rules” as those described above – set questions, with the freedom to ask others, length determined by the interviewees’ interest (within reason – some interviews would last for days if the interviewer allowed them to), and open-ended and follow-up questions. The big difference is that people in a small group play off one another, asking their own questions, volunteering follow-up answers before the questions are asked, and elaborating on others’ points (as well as providing that check on others’ accuracy mentioned above).

Focus groups are more purposeful small-group interviews.  Here, the questions are determined beforehand, and the facilitator/interviewer is trained to coax out as much information as possible.  In general, focus group participants aren’t told exactly what information the facilitator is looking for (although that’s not always the case in a health and community service context), and participants often don’t know one another. The reason behind both these restrictions is to try to get as unbiased a reaction as possible from everyone in the group.  If they don’t know exactly what the interviewer wants to know, they can’t alter their comments to please him (or annoy him); if they don’t know anyone else in the group, they’re less likely to play their accustomed roles, or to edit their comments so that they won’t be repeated to other friends and acquaintances.

Town-meeting or whole program format. In this situation, a large gathering of people – perhaps even all participants in a program – meets together to provide feedback.  How good a method this is depends upon the circumstances. If it’s facilitated well, a town-meeting format, where individuals can speak their minds and where give-and-take is welcomed, can be a generator of useful feedback and new ideas. It can be used as a brainstorming session, it can generate volunteers to take on the tasks that grow out of the feedback provided, and can lead to an explosion of involvement and creativity from participants and staff. If it’s not well facilitated, or if there are serious issues that haven’t been dealt with, it can degenerate into name-calling and anger, or produce little more than silence. It’s hard to decide which is worse.

All this is to say that a town-meeting format can be either a dream come true or a nightmare. If you want to use it, make sure that (a) you have a good facilitator who understands the program and its situation, and who knows how to steer discussion in productive directions; and (b) there isn’t a major problem that has gone unattended, or an individual or group with an axe to grind that wants nothing more than to use this meeting as a grindstone.  In those instances, you may get lots of feedback, but it may produce more bad feeling than you know what to do with.

Journals. A way of obtaining ongoing feedback is to ask participants to keep journals. These may be devoted solely to feedback, or may have a larger programmatic purpose, depending on the nature of the program. Since people write in them on a regular basis – daily, weekly, or some other period – journals can provide a running commentary on participants’ reactions to the program, to individual parts of it, to events, etc.

If the journals are interactive – participants’ journal entries are read and responded to in the journal by the educator or other relevant staff member (a counselor, a mentor, a health professional) – participants can be asked to elaborate or expand on particular comments, and the feedback can be extremely focused and valuable. This method only works, of course, if participants are able and willing to keep a journal, and if they trust the staff member enough to reveal their real thoughts and feelings about the program.  It’s time-consuming, but if the time exists – or if the journals serve another purpose as well – it can be extremely helpful.

There are, of course, other ways of obtaining feedback.  The editor of this section, for instance, recalls a mental health center that asked people to evaluate the quality of the therapy they received.  At the end of each session, they were encouraged to grab a poker chip from a dispenser at the front desk and deposit it in a box – blue for good, white for okay, red for poor.  It was simple, anonymous, and informative.

Which method you decide to use will depend on a number of factors.  If time is an issue, for instance, you might not want to use individual interviews or journals. If you want detailed feedback, including opinions and the opportunity to follow up on people’s ideas, on the other hand, some sort of interview is probably the best way to get it.  A town-meeting format may be particularly appropriate in a situation where large gatherings are a regular part of the organization’s operation. Surveys might work best where participants are scattered geographically, or where you’re polling former as well as current participants.  Journals are particularly appropriate where they’re part of the program (a parenting group or adult basic education class, for instance).

Ways to improve your chances of getting good feedback

Regardless of what methods you use to gather feedback from participants, there are some things you can do to increase the chances that the information and opinions you get will be honest, and meant in the spirit in which they’re requested.  Not all of these are possible for all methods of soliciting feedback, and not all of them are necessary or appropriate for a given group of participants.  It’s up to you to determine what might work best for your effort.

Let people know from the beginning that you want their feedback, and will ask for it

This alerts them that they should be thinking about what they approve and don’t approve of, enlists them as partners in improving the enterprise, and gives them “permission” to look at your effort critically.  You might also explain that you want ongoing feedback, and that their opinions and ideas will be welcomed whether they’re directly requested or not.

Specify what kind of feedback you want

The more definite you are about the kind of information you’re seeking, the more likely you are to get it.

Some of the questions you might ask or things you might want to know about:

Assessment / planning:

• What’s the need for (your particular service) or importance of (your particular issue) in the community?  Is there a group that’s most in need of/affected by it?
• Has this service/issue been addressed in the community before?  What was the result?
• Who are the most important people in the community/a specific population to talk to about or try to get involved in this effort?
• Are there people on that list who simply can’t be in the same room together?  Who work really well together?
• What do you think might draw people like you into this effort/program?  Are there things we absolutely shouldn’t try?
• Please let us know if anything that’s suggested in the planning process is culturally offensive, or simply doesn’t fit into the way you do things

Ongoing feedback and program evaluation:

• What do you like about the program?  What works well for you?
• What would you change about the program?  How?  What doesn’t work well for you?
• What would assure that you’d stay in the program?  (What did keep you in the program?)
• What would absolutely make you leave the program?  (What did make you leave?)
• Was/is intake welcoming?  Informative – did you get all your questions answered?  Respectful?  Relatively easy (determining eligibility, for instance)?  Did the intake people try to make it as easy and pleasant as possible?
• Were you placed at the right level?  Was placement flexible?
• Are you getting the services you were promised?  Are they the services you need?  Do you need services that this program provides that you’re not getting?
• Do you need support services that aren’t available (child care, transportation, accessibility, etc.)?
• Are staff supportive, helpful, and respectful?  Do they seem to know what they’re doing?
• Are you happy with the way you’re treated?  Are you being treated as an equal, with respect?

This question is really about the philosophy of your organization, and its view of participants.  Staff people may be polite, but if their attitude toward participants is that they’re incompetent to run their own lives, that will be communicated, courtesy or no courtesy.  By and large, people respond positively to being treated as equals, with respect, and resent being treated as inferiors.  Part of that respect may be honesty about hard truths (“You really screwed up here.”), but honesty doesn’t prevent a basic assumption of equality. Taking participant feedback on this subject seriously can be hard for an organization.  You may feel you’re doing your best, and getting very little recognition from participants for your efforts.  But your best efforts at providing service can be derailed if they’re not accompanied by an attitude of true respect toward those with whom you’re working.  If you’re hearing that participants aren’t feeling respected, it may be time to honestly reassess the philosophy and guiding principles of your organization.

• Are program expectations clear and consistent?  Do staff people seem to care?  Do you get help when you need it?
• How do you get along with other participants?  Do you enjoy the social situation?  If there are problems, have you spoken to anyone (staff or other participants) about them, and are they getting addressed?
• Has there been any problem about payment (or insurance or eligibility)?  Is it easy to take care of?  Are you comfortable talking to staff about any problems with it?
• Does the schedule work for you?  What would you need to make it better (evening or weekend hours, for instance)?
• Is the location one that means a lot of travel for you?  Is there convenient public transportation?  Does the neighborhood/area feel safe?
• Is the program site comfortable?  Does it serve the purpose of the program well?  Is it big enough, and are there enough separate rooms?  Is it accessible for you and anyone else who needs to use it?  Are the restroom facilities adequate?

Make clear exactly what you’re going to use the feedback for

Whether it’s assessment, planning, ongoing adjustment, or evaluation, let people know what you’re going to do with what they give you.  Some participants might feel that their feedback will be used to judge them, or that someone will be fired if they say anything negative or suggest changes.  If you can explain clearly that the purpose of the feedback is to improve the program and make sure it’s doing what it’s supposed to, and what participants need – and that no one will be fired, but rather that everyone will be helped to do his job better – that may remove some of their reluctance to respond.

Openness about what you’re asking for also fits into the theme of participatory process, enlisting participants as planners or evaluators, and helping them to see their opinions as valuable, and themselves as partners in your effort.

Guarantee anonymity

Many of the reservations or fears participants may have about being negative or straightforward can be overcome if they know that no one will know that they are the authors of their comments.  Written surveys, particularly those that require only circling or checking off answers, are obviously the easiest to hold to this standard.  Those filling them out can drop them off in a neutral spot – a box in the office, for instance – with nothing to identify them as connected to a particular form.  Those participating in face-to-face or phone surveys or individual interviews have to depend upon the promise of the interviewer that their identity won’t be attached to their comments or revealed to anyone.

Where it is obvious, for cultural or other reasons, that fear of being identified is an issue, you might consider using a method that can guarantee anonymity.  Small or large groups pose much more of a problem in this area, but you can sometimes skirt the issue by having a group facilitated by someone unconnected with the organization – a volunteer or outside facilitator, for example.  (This might work if people’s reluctance to be identified only extends to program staff.  If it includes other participants, the group situation would still be a problem.)

Let people know that you’ll share the results of feedback with them, and be sure to do it

Participants should expect that you’ll tell them the overall results of the feedback you got.  You should also explain how you’re using it, and get their feedback on that.  That further cements the partnership, as well as giving participants the opportunity to react if you’re changing a part of the program that most people like, or ignoring what most see as a problem.

How do you use feedback from participants?

So now that you’ve gotten all this feedback, what are you going to do with it?  The last thing you want to do is to solicit feedback, get it, and then file it away and never look at it again. The whole point is to use it to improve your work.  Good feedback presents you with a broad range of opportunity.

Understand the needs of the community

Feedback from potential participants and the community before you start planning can show you what the real needs and concerns of the community are, what the appropriate levels of service are, and what issues you should concentrate on.

Understand changes in the community, or in the situation of the participant population

Increased immigration, for instance, can have a significant effect not only on the community as a whole, but on the resident population – Cambodians, Hispanics – to which the immigrants belong, and can change the needs or the concerns of that population. A factory closing, the start of a major construction project in a particular neighborhood, gentrification – all of these can cause major changes for participants, and require that you alter the approach or the content of your effort.

You might or might not be aware of these community changes themselves, and participants can obviously tell you about them. The real value of participant feedback here, however, is in what those changes mean for the folks you work with. Their feedback can tell you how to respond to changes in order to keep your program on track, and to make sure it’s as useful as possible to the people it serves.

Improve your program itself

Participant feedback can be tremendously useful, both ongoing and as part of an evaluation, in helping you make your program more effective.

In terms of the actual work of the program, it can alert you to the need to refine your techniques and methods, beef up what’s working, and restructure what’s not. It can let you know when program content isn’t relevant, and has to be rethought. Participant feedback can also help you realize when you need to change schedules, try to provide support services, and otherwise make your program more user-friendly.

The other area where participant feedback can have a real impact on program effectiveness is that of program philosophy.  As we’ve stressed, what participants say about how they are treated and how they feel about coming can be extremely important.

 There are really three areas to look at here, all of which have already been referred to in this section:

• Attitude toward participants. If participants aren’t treated with respect, they’re less likely to remain in a program, and less likely to have success in it if they do.
• Approach toward provision of services. When participants are accepted as partners and feel ownership of a program, they’re not only more apt to get a lot out of it, but to contribute to its overall success as well.
• Identifying and addressing barriers to participation or success. Part of being respectful and treating participants as partners is responding creatively to participants’ needs for support services, particularly child care.  This attitude also increases the chances that program staff will realize when, for example, a participant in an employment training program has basic skills needs, or simply can’t see well enough with her present glasses to perform necessary tasks.  Barriers like these can be addressed only if they are perceived.

Develop an organizational culture that embraces and institutionalizes ongoing feedback from participants

So many of the “how-to” portions of Tool Box sections end with the advice “And then keep at it forever.” This is the equivalent of that advice for obtaining and using participant feedback.

Feedback from participants isn’t a one-time thing.  It has to be encouraged continually: participants and staff come and go, conditions change, and institutional memory – the knowledge of what has happened in an organization over time – can be short. An organization, a program, an intervention must continually reinvent itself in order to continue to effectively meet the real needs of those it serves. Without ongoing participant feedback, that reinvention is unlikely to happen, and even less likely to have any real impact if it does.

In Summary

Feedback from those who have the closest perspective on what you’re doing – those you serve, or whom your work is meant to benefit – can be tremendously helpful in making your program, initiative, or intervention as effective as possible.  It’s not always easy to get honest feedback from these folks, because they may be reluctant to criticize, or may be afraid to respond at all for various reasons.  But their feedback is important enough that it’s worth the effort to get around their reluctance.  Perhaps the best way to ensure honest feedback is to involve participants in what you’re doing from the beginning.  If they’re part of the planning, implementation, and evaluation of your work, they’ll feel ownership, and trust you and the work enough to tell you what they really think.

Participant feedback can help you better understand what you need to do and how to do it. It can improve the work you’re already doing, and make sure you adjust to changes in the community or the population that call for changes in your approach or your goals. Overall, the purpose of obtaining feedback is to assure the quality of your work.

You might obtain participant feedback through the use of surveys (written, face to face, or by phone), individual or group interviews, focus groups, town-meeting or whole-program format meetings, or journals. Whatever method you choose, you’re more likely to get honest feedback if you treat participants as partners working with you toward a common goal, rather than as clients who couldn’t function without you.

Ultimately, you should strive toward creating an organizational culture and attitude that encourages participant feedback, and views it as a necessary tool to keep increasing the quality of the work you do.

What do we mean by feedback from participants?

___Feedback from participants consists of reactions to, opinions about, and/or information on what you do from those to whom it’s offered or who benefit from it

Feedback may be:

___Positive, negative, or neutral

___Objective or subjective

___Enhanced by knowledge of community history, personalities and relationships, culture, etc.

___Limited by lack of knowledge, information, or understanding

___Solicited or unsolicited

___Direct or indirect

Why do you need feedback from participants?

___It’s part of being customer-centered

___It gives you tools to improve your program

___It allows you to respond to changes in the community, the population, or the situation

___It can give you information about the history of the community, the history of your issue in the community, or the history of your population

___It can inform you about personalities and relationships in the community and/or the population you’re working with

___It can tell you when your methods or approach aren’t working

___It can tell you when you’re treating people in ways that make them feel uncomfortable, angry, or otherwise disrespected

___It can help you deal with what’s important to participants, even though that may not be your major goal

___It can assure that your objectives make as much sense for participants as they do for you

Who are participants that can offer feedback?

___Adults and teens with no intellectual, psychological, or social barriers

___People with mental illness

___Children

___At-risk or troubled youth

___Developmentally delayed or retarded teens and adults

___Speakers of other than the majority language

___People who have left the program

When might you ask for feedback from participants?

___Before you begin an effort (Plan)

___As you develop a program, initiative, or intervention (Do)

___Before you implement your plan (Check)

___While the work progresses and in evaluating it (Act)

___In making adjustments and starting the cycle again (Analyze)

What are some methods for obtaining feedback from participants?

___Surveys

___Individual interviews

___Small group interviews and focus groups

___Town meeting or whole-program format meetings

___Journals

How can you increase your chances of getting good feedback?

___Guaranteeing anonymity

___Explaining why you need feedback, and how you’ll use it

___Being clear about exactly what kind of feedback you want

___Sharing the results with participants

How do you use feedback from participants?

___Use it to understand the needs of the community

___Use it to respond to changes in the situation of participants, the community, the political climate, etc.

___Use it to improve your program by responding to the need for change in attitudes, methods, content, etc.

___Use it to create or strengthen an organizational culture that encourages and welcomes feedback, and uses it well

 

Examples of oversight mechanisms include:

• A citizen review board that investigates complaints of police brutality.
• An in-house quality control team that determines whether the company’s methods and products are up to standard.
• A system by which students can evaluate their courses and the teaching of their instructors.
• A diverse committee that reviews the experiences of patients at a local clinic.

In this section, we’ll look at ways to set up processes that help assure accountability.

What do we mean by oversight and oversight mechanisms?

Oversight refers to maintaining a watchful eye on something. For instance, in the case of a citizen review board, an internal group monitors the quality and effectiveness of the service their organization provides. Oversight involves taking responsibility for the operation, implementation, and/or outcome of a process, program, organization or institution, or effort.

An oversight mechanism is the system or process used to maintain a watchful eye.

Just as every writer needs an editor, those doing work that affects the community need monitoring to assure that their work is effective, of high quality, and follows rules and regulations properly.

Oversight is a matter of someone paying attention. It is meant to catch potential problems as they arise, and to make sure that nothing is being done that’s likely to cause problems later. For instance, good oversight can detect illegal or unethical practices on the part of a business or government agency, pinpoint and change bad financial practices in a non-profit organization, or make sure that the public’s health is protected.

Oversight may be official – required by law or by external or internal regulations – or unofficial, as in the case of a watchdog group overseeing whether a government agency conducts itself fairly and according to its mandate. When oversight is official, the mechanisms for it – as well as the individual or group that is responsible for exercising it – are often specified by the laws or regulations that require it. When oversight is unofficial, the overseeing group or individual may be self-appointed, or may be asked to do the job.

Where oversight involves responsibility for the functioning and/or results of something, it can be set up within an agency, corporation, institution, or organization, or may be overseen by another body.

The activities of community health and development organizations can span a broad range, from planning, to intervention, to advocacy, among others. Maintaining the quality of these activities requires oversight mechanisms that reflect the vision, mission, values, and goals of the groups and individuals engaging in them. We’ll discuss establishing oversight mechanisms with that in mind.

Why establish oversight mechanisms?

There are a number of reasons why you might want to set up oversight:

• To assure accountability. Oversight is also used to hold individuals and groups accountable for operating legally, ethically, and responsibly, and for producing the services, products, and/or outcomes they have promised or are contracted to provide.
• To control quality. We normally think of quality control as an industrial issue, but it also applies to the quality of services, facilities, etc.
• To ensure adherence to laws and regulations. Oversight may be used to make sure those agencies, corporations, and other entities – or departments or programs within those entities – are operating according to the rules set up by laws, controlling agencies, funders, or the entities themselves.
• To control unfair treatment, corruption, illegality, and unethical behavior. Oversight includes processes for protecting against unfair treatment in which some groups benefit and others are excluded. Corruption by public officials—including the theft or misdirection of government funds and goods—can often be prevented, detected, or stopped by careful oversight. The same is true for illegal or unethical acts that are harmful to society, to particular groups, and to those who are meant to be served or helped by the groups or individuals committing those acts.
• To ensure the effectiveness of programs or efforts for which you have responsibility. If you’re responsible, you need to make sure that everyone involved is following the process that’s been set up, and that the work is going according to plan. Oversight will also help you spot when the plan isn’t working, and help you and others involved determine how to change or fix it.
• To help improve the wellbeing for everyone. Community wellbeing is enhanced when community interventions and initiatives are effective, when government is doing its job, when employers treat workers with respect and with concern for their safety, when business and government work to preserve rather than damage the environment, and when all community members get what they need to live.

Who should be involved in establishing oversight mechanisms?

The goal of effective oversight is to improve quality and to help people and organizations do the best work they can on behalf of the community. It often makes sense, therefore, to involve a variety of people and groups in the oversight process. Asking participants or students to evaluate the services or education they’re getting and recruiting community members to oversee local government services are two of many ways to accomplish this.

Those who might be involved in oversight to make the process more comprehensive and useful include:

• Those from the groups most affected by the issue or situation involved.
• Those providing the service or products for which oversight is directed (i.e., those being overseen).
• Those ultimately responsible for the success of the effort – program directors, school principals, elected or appointed officials, etc.
• Funders and others contributing resources or support to the implementation of the effort.
• Community activists concerned with the issue or effort in question.

The oversight process works best if it is participatory, including both those affected and those responsible. Selection of who participates can be more challenging when the group or individuals at whom oversight is directed are resistant to being watched.

When should you establish oversight mechanisms?

The obvious answer here is “whenever oversight is necessary.” But when is that? The short answer is “whenever there is an effort or situation that needs attention.” Let’s examine what that means.

• When an effort or new program or organization is starting. Oversight should be ongoing from the very beginning, so that quality is at the front of everyone’s mind, and so that problems with implementation or unfair treatment can be caught and addressed as early as possible.
• When a problem needs to be prevented from occurring again. For instance, if the community has experienced patterns of police brutality or discrimination in housing, one way to address the situation is to set up a citizen review committee. This can help address unfair treatment by assuring accountability and establishing a channel of communication between community members and those responsible for institutions that should serve them.
• When the quality or effectiveness of an effort or program is in question. In the ideal, there would have been oversight at the beginning; but if that wasn't the case, it should begin now, and continue as long as the effort does.
• When there is an agreement, contract, or regulation to be followed. Oversight can ensure that all parties fulfill their obligations and get whatever compensation or benefits were promised.

Oversight in this case is often internal, in order to make sure, for example, that money is spent in the ways agreed upon with the funder. When oversight lapses, or when there is none, organizations may misspend money and be forced to pay it back or to close.

• When an entity is engaged in an activity that is potentially harmful to the community, its members, or society. This can mean a violation of the law, such as illegal dumping of pollutants. It can also, however, refer to a private or government-funded construction project, a water treatment or food processing operation, or other perfectly legal activity that nonetheless could cause harm if procedures were not followed. Oversight can both help to prevent harmful situations, and to quickly catch them and spur appropriate steps if they do occur.

How do you establish oversight mechanisms?

In most situations, gather a group of stakeholders to plan and set up oversight mechanisms. As we’ve discussed, oversight often works best when it’s welcomed by everyone involved, and seen as a way to improve the work or the situation. One possible exception here is when the oversight is meant to catch suspected illegal, unethical, or otherwise harmful activity on the part of an entity. Even then, depending on the circumstances, it may make sense to involve representatives of the entity being overseen.

Decide what kind of oversight you need. The nature of the oversight will help you to structure an effective oversight mechanism. There are many different kinds of oversight, including:

• Oversight in terms of accountability. Key questions include: Are people and entities doing their jobs? Do programs and initiatives have a positive effect (and do they have the intended effects)? Who’s responsible for making things better, and who needs help? Are government bodies implementing programs effectively, providing necessary public services, and enforcing laws and regulations consistently and fairly?
• Oversight of the operation of a program, effort, or organization. Key questions include: Is it going according to plan? Is it using the methods and procedures it was designed to? Is it accomplishing what it’s meant to do?
• Oversight of conduct of a service, organization, government agency, etc. Is there fairness, discrimination, or consistency of quality? Is it adhering to rules and regulations in a reasonable way (compliance)? Is it doing its job?
• Oversight of the performance of individuals in an organization or institution – supervision of service providers, teachers, managers, etc. This works better as a formative evaluation; that is, instrumental in helping the individual improve performance—rather than summative judgment leading to punishment.
• Oversight of the conduct of individuals in organizations and government. Are they adhering to the rules, behaving ethically, not overstepping their authority, not engaged in conflicts of interest, etc.?
• Oversight of processes to make sure they’re conducted properly, and that policies and procedures are followed and effective. This includes oversight of budgets to ensure that no one is overspending, or over- (or under-) spending in a particular area, and to flag potential fiscal problems in time to address them.
• Oversight of agreements and contracts to be sure that their terms are carried out as specified, and that both parties are fulfilling their obligations and receiving what was agreed upon.
• Oversight to protect vulnerable individuals. This refers to certain individuals, as well as the services provided to them, that may need to be monitored for their own safety or well-being; for instance, a child with special needs, a hospital or clinic patient, or a frail elder.

Choose an oversight mechanism that is best suited to the type of oversight you expect to conduct. Oversight mechanisms can take a number of different forms, depending on their purpose:

• Oversight by a designated or responsible individual or group. Oversight of an organization may be conducted by its director. Fiscal oversight of an organization, institution, or business may be conducted by its chief fiscal officer, treasurer, or a board financial committee. In some cases, certain individuals or groups – often boards of directors – are legally responsible for ensuring that their organizations act legally and in compliance with the requirements of funders and contracts.
• External oversight by a funder, government agency, or other responsible body. This kind of oversight may take the form of spot checks or regular reports (see below), or may be contracted out to a consultant or third party.
• An oversight committee. This might be a citizen oversight panel for a government agency or service such as the police, environmental services, or public health; or an internal oversight committee within an agency, institution, corporation, or organization. Oversight committees are generally official, established either by or with the collaboration of the body that’s being overseen. Citizen oversight committees almost always include members of the population most affected by the entity being overseen; for instance, tenants for a public housing authority or community residents for a local community health board.
• A watchdog group. This is an organization or less formal group whose mission is to identify and publicize corruption, lawbreaking, discrimination, or other illegal or unethical behavior on the part of government agencies or officials, business and industry, or other sectors of the community that have the power to harm individuals, groups, or the community or society as a whole.
• A monitoring and evaluation process. This could be internal or external, with the monitoring and evaluation carried on within an organization or institution by staff members, or conducted by a consultant or other body from outside. In the ideal, evaluations might be produced annually or more often. These could then be used to assess progress in meeting objectives or determine the quality of services, and to use that information to make indicated improvements.

There are many ways to conduct an evaluation process. It can be collaborative, with everyone involved contributing; or it can be carried out solely by an individual or small group with limited input from anyone else. The Tool Box recommends that an evaluation process be as collaborative and inclusive as possible. If it’s devised and owned by those being overseen by it, and if they participate actively in it, it’s more likely not only to go smoothly, but also to produce accurate and useful information. If it’s seen as an adversary process, an evaluation can precipitate withholding information and cooperation in ways that reduce utility.

• Spot checks by staff, supervisors or program directors. This method might be used in almost any oversight situation where scarce resources make more intensive oversight difficult.
• Regular oral and/or written reports on the work being done and its results. Since the reports are usually prepared by those doing the work that’s being overseen, the effectiveness of this mechanism depends on the reporters’ honesty, perceptiveness, and critical reflection on what is happening.
• Mechanical oversight, as by the use of cameras, audio recorders, or mechanical inspection of activities and products. This method might be used for supervision of activities (e.g., client service) or for environmental efforts (oversight of deforestation or biodiversity, for example). In most cases, the results of the mechanical oversight should be available to review by those who performance or outputs are being watched.

Any or all of these mechanisms can be enhanced by feedback and systematic reflection on what is occurring by those who are the objects of the oversight, and from those most affected by the entity or organization responsible.

Decide who will conduct the actual oversight. This will be determined, at least in part, by what kind of oversight you’re conducting. Internal monitoring might be done by supervisors, with the cooperation of the staff members being monitored, or by a committee composed of staff from different parts of the organization. A funder might exercise oversight through an external consultant or evaluator, or through reporting by the funded agency or organization. An organization’s treasurer or fiscal officer might oversee the budget to make sure that funding is being spent for the approved purposes. A community oversight committee would be composed of community members affected by the agency or activities in question.

Decide on the methods you will use to conduct the oversight. They should be appropriate to the situation and to the character of your organization or effort. The more your organization emphasizes inclusion and participation, for example, the more inclusive and participatory your oversight should be. Any oversight you conduct should model your organization’s vision, philosophy, and standards.

Obviously, methods can vary greatly, depending on the nature of the oversight. You wouldn’t oversee an environmental improvement operation – forest management, wildlife habitat restoration – in the same way you would the functioning of a hospital or the conduct of a government agency.

​Some possible methods you might use to implement oversight mechanisms:

• Individual and group interviews with administrators or managers and staff members, present and past participants, funders, and others affected by or familiar with the entity’s work.
• Casual conversation with staff members and administrators.
• Direct observation of the work of the entity. This might entail accompanying staff members periodically, walking through a facility and stopping to watch activity, staying in one place to observe action in the area, etc. In some cases, it might mean going to a place at many different times, days of the week, and seasons to understand what happens there under different circumstances.
• Embedding into the work of the entity, also known as participant observation. In this situation, the person conducting oversight takes on the role of a staff member or administrator and participates in the day-to-day life of the entity.
• Study of organizational or agency files, records, and financial reports, as well as its history, funding sources, and philosophy.
• Research into practice models or theory that underlies the work of the entity.

Train those who will conduct the oversight if necessary. Conducting oversight can be different from conducting an evaluation. You may or may not have the cooperation of the individuals or entity being overseen, even if they are involved in establishing an oversight mechanism for the process. Some important guidelines for oversight, therefore, concern how you approach the individuals and entities subject to it.

You, or whoever is involved in conducting the oversight, particularly those who have never done something similar before, may need training. Some things those who actually conduct the oversight should be guided by include:

• Get to know the organization or entity you’re overseeing – its history, structure, vision and mission, finances, the nature of its staff and administrators, the organizational culture, etc. Learn as much as possible about the work of the entity and the individuals who perform it. Try to get a sense of the pressures and requirements of the work, and of the commitment staff members have to it and to their employer.
• Understand that people may not want you there, either because they don’t want oversight, or because you’re interrupting their work.
• Try to establish a rapport with people you interview or interact with. Be natural and respectful. Treat everyone like a colleague.
• Listen carefully to what people tell you, and ask follow-up questions where it’s appropriate. Listen more than you talk.
• Use open-ended questions as much as possible, i.e., questions that require more than a yes-or-no answer.
• Whenever you’re engaging in oversight activities, use all your senses. Watch body language, pay attention to the environment – noise level, smells, temperature, etc. – and to the interactions among people.
• Take careful notes in whatever way is easiest and most comfortable for you and others. That may mean pencil and paper, a laptop or notebook computer, a smart phone, or a small tape recorder. Whatever you use, you should explain what you’re doing to anyone you have formal contact with. If you take pictures, or record via audio or video, you should get people’s permission beforehand.
• Respect people’s time. Be on time for meetings and appointments, and don’t take more time than necessary.
• Produce reports regularly, and share your findings with those who are subject to oversight.
• Praise people for their work where appropriate. Don’t blame or criticize. Your findings should make clear where there need to be improvements.

Create an analysis and reporting system. Both the process and the observations that come out of oversight should be recorded for two reasons:

• To show that the organization, institution, or effort is doing its job in keeping track of what goes on.
• To use the information to improve both oversight and the quality of the operation.

To accomplish both these purposes, you’ll need to regularly record and make available the information you get from oversight mechanisms. In addition, you’ll want to develop a process for analyzing that information so that you can spot potential problems or irregularities and correct them.

Catching potential problems with finances is particularly important area for oversight. In the spring of 2012, J.P. Morgan-Chase, one of the largest banks in the U.S., and supposedly the best-managed, announced a loss of over $2 billion that occurred because the company’s top-level management and Board of Directors failed to exercise proper oversight of the bank’s investments. While most non-profit organizations are not likely to lose billions to bad investments, they can nonetheless make financial decisions that lead to limiting services, laying-off staff members, or going out of business entirely. If the right people in the organization get accurate information from proper oversight, they can prevent many of these failures or consequences. That’s why reporting and analysis are necessary.

Determine how you will use oversight to maintain or improve the quality of your work or of the entity or situation you’re overseeing. What areas of the work or organization will you oversee? This question is especially important if you don’t have a lot of resources to put toward oversight. Another important issue is what purpose your oversight will serve. Is it meant to prevent something – harassment or discrimination in the workplace, illegal pollution of the air or waterways, misuse of public funds – or is the purpose to maintain a standard of practice and effectiveness? Is it meant to show – accurately – that you’re practicing “due diligence” (i.e., doing what you’re supposed to be doing) by keeping careful track of all phases of your effort or organization? How will you use the information you get from oversight to analyze your work and change it to improve it?

The answers to these questions will help to determine how to set up an oversight mechanism that meets your needs. If you’re using oversight to prevent illegal or unethical activity, for example, you probably won’t announce when you’re going to be watching. The opposite may be true if part of its goal is to improve practice or the functioning of an organization or system. If your goal is to keep careful track of finances or of another aspect of an organization or agency, then you’ll want to set up procedures to make that as easy as possible to see patterns in income and expenditures.

Evaluate the effectiveness of your oversight mechanism and adjust it as necessary. Oversight itself should be monitored to make sure that it is attending to everything that needs attention. If it’s not, the mechanism, like any other aspect of your work, should be adjusted to make it work well. Checking regularly – at least every few months – to make sure you are getting the information that allows proper oversight can save a great deal of hassle in the long run.

Continue oversight over the long term. As we explained at the beginning, oversight is a long-term commitment. If things are going well, exercising good oversight helps to maintain progress. If oversight is meant to prevent problems, you can bank on problems arising as soon as oversight stops. You need to establish oversight mechanisms that can be sustained indefinitely, and then use them consistently to make sure that all goes well.

In Summary

We set up oversight mechanisms to keep a watchful eye on things. Most organizations, institutions, and businesses – and some governments – exercise some form of oversight over their own operations to make sure they don’t make financial, legal, ethical, or programmatic mistakes that can threaten their effectiveness or existence. In addition, funders, government agencies, and other entities may exercise oversight over those to which they give resources or for which they are legally responsible. Citizens or citizen organizations may oversee entities that affect their lives or that can harm individuals or the community. For public agencies, having a board of directors may be a legal obligation.

Oversight mechanisms often work best when both those who employ them and those subject to them participate in their development and implementation. They need clarity of purpose and structure, consistent methods, and training where necessary for those who will use them. Technical support can be helpful in establishing analysis and reporting systems, regular monitoring, and using the information to make adjustments.

 

We encourage the reproduction of this material, but ask that you credit the Community Tool Box: /.

Contributor

Phil Rabinowitz

Editors

Andrea Glinn

Stephen Fawcett

Here you will find a checklist summarizing the important points of the section.

 

What do we mean by oversight and oversight mechanisms?

­­__ Oversight is the maintenance of a watchful eye on an individual, entity or process.

Oversight may also be the responsibility for the operation, implementation, and/or outcome of a process, program, organization or institution, or effort.

__ Oversight may be internal or external.

__ Oversight is different from evaluation in that it is constant and concerned with proper functioning as much as effectiveness.

__ An oversight mechanism is the system or process and the procedures used to exercise oversight.

__ Oversight and oversight mechanisms in health and community services and development should reflect the vision, mission, values, and goals of the groups and individuals engaging in them.

 

Why establish oversight mechanisms?

__ To assure accountability.

__ To control quality.

__ To ensure adherence to laws and regulations.

__ To control unfair treatment, corruption, illegality, and unethical behavior.

__ To ensure the effectiveness of programs or efforts for which you have responsibility.

__ To help improve the wellbeing of everyone.

 

Who should be involved in establishing oversight mechanisms?

__ Where possible and appropriate, establishing oversight mechanisms should be a participatory process, and involve:

• Those from the groups most affected by the issue or situation involved.
• Those providing the service or products for which oversight is directed (i.e., those being overseen).
• Those ultimately responsible for the success of the effort – program directors, school principals, elected or appointed officials, etc.
• Funders and others contributing resources or support to the implementation of the effort.
• Community activists concerned with the issue or effort in question.

 

When should oversight mechanisms be established?

__ When an effort or new program or organization is starting.

__ When a problem needs to be prevented from occurring.

__ When the quality or effectiveness of an effort or program is in question.

__ When there is an agreement, contract, or regulation to be followed.

__ When an entity is engaged in an activity that is potentially harmful to the community, its members, or society.

 

How do you establish oversight mechanisms?

__ In most situations, gather a group of stakeholders to plan and set up oversight mechanisms.

__ Decide what kind of oversight you need. Possibilities include:

• Oversight in terms of accountability.
• Oversight of the operation of a program, effort, or organization.
• Oversight of the conduct of a service, organization, government agency, etc.
• Oversight of the performance of individuals in an organization or institution – supervision of service providers, teachers, etc.
• Oversight of the conduct of individuals in organizations and government.
• Oversight of processes to make sure they’re conducted properly, and that policies and procedures followed and effective.
• Oversight of agreements and contracts to be sure that their terms are carried out as specified, and that both parties are fulfilling their obligations and receiving what was agreed upon.
• Oversight to protect vulnerable individuals.

__ Choose an oversight mechanism that is best suited to the type of oversight you expect to conduct. Some types of oversight mechanisms:

• Oversight by a designated or responsible individual.
• External oversight by a funder, government agency, or other responsible body.

• An oversight committee.
• A watchdog group.
• A monitoring and evaluation process.
• Spot checks by staff, supervisors or program directors.
• Regular oral and/or written reports on the work being done and its results.
• Mechanical oversight, as by the use of cameras, audio recorders, or mechanical inspection of products.

__ Decide who will conduct the actual oversight.

__ Decide on the methods you will use to conduct the oversight.

Some possible methods you might use with various oversight mechanisms:

• Individual and group interviews with administrators or managers and staff members, present and past participants, funders, and others affected by or familiar with the entity’s work.
• Casual conversation with staff members and administrators.
• Direct observation of the work of the entity.
• Embedding into the work of the entity, also known as participant observation.
• Study of organizational or agency files, records, and financial reports, as well as its history, funding sources, and philosophy.
• Research into practice models or theory that underlies the work of the entity.

__ Train those who will conduct the oversight if necessary. Some possible topics:

• Get to know the organization or entity you’re overseeing.
• Understand that people may not want you there, either because they don’t want oversight, or because you’re interrupting their work.
• Try to establish a rapport with people you interview or interact with. Be natural and respectful.
• Listen carefully to what people tell you, and ask follow-up questions where it’s appropriate.
• Use open-ended questions.
• Pay attention to body language, the environment, and to the interactions among people.
• Take careful notes in whatever way is easiest and most comfortable for you and others.
• Respect people’s time.
• Produce reports regularly, and share your findings with those who are subject to oversight.
• Praise where appropriate, but don’t blame or criticize.

__ Create an analysis and reporting system.

__ Determine how you will use oversight to maintain or improve the quality of your work or of the entity or situation you’re overseeing.

__ Evaluate the effectiveness of your oversight mechanism and adjust it as necessary.

__ Continue oversight over the long term.

• What do we mean by formal public reporting processes?

• Why create formal public reporting processes?

• Who should be involved in creating formal public reporting processes?

• When should you create formal public reporting processes?

• How do you create formal public reporting processes?

Community efforts are just that – they work best when a large part of the community is involved in some way. Not everyone can be on a planning or oversight committee, or take part in implementation, however. So how do you keep the community involved and supportive?

One obvious way is to assure that individuals/community members are informed about priority community issues, about efforts to address them, and about the final/end result. This section will help you develop communication and reporting processes that keep the community informed and involved, and provide accountability for the effort.

What do we mean by formal public reporting processes?

Reporting can just be a matter of telling people something. You might bump into a friend and report to her about a meeting you attended or a class you took. These types of occurrences represent informal reporting.

A formal reporting process generally has specific characteristics:

• Formal reports must be routinely delivered. Recipients should expect to receive regularly scheduled status updates, such as quarterly and/or annual progress reports. Reporting must be based on consistent measures, which may be quantitative (number-based), qualitative (narrative, interviews, or other elements that cannot easily be translated into numbers), or both. It should be noted that these methods should be consistent throughout each respective report to ensure that it is possible to conduct comparative analyses.
• Formal reports must offer suggestions for adjustment and change and continue throughout the life of the effort.

Why create formal public reporting processes?

• To involve the community and garner support. Formal reporting processes keep the community informed and invite community members to join the effort, making it possible for them to feel ownership of and responsibility for the effort. In addition, involving community members in devising these processes further ties them to the effort.
• To bring the issue to the fore and raise the profile of the effort. Advertisement may prove fruitful when marketing the effort and engaging those who were previously unaware or unfamiliar with the initiative.
• To hold the effort accountable. It is important to not only publicize the effort’s achievements throughout the community, but to also recognize and communicate adjustments that must be made in order to improve the initiative’s effectiveness for positive outcomes.
• To keep track of progress. Routine reporting is essential so that both the public and those involved in implementation have a clear understanding of the initiative’s work and accomplishments.
• To collect new ideas for the effort. Community meetings and other means of disseminating information about the effort allow for public feedback, which may be extremely useful in adjusting the work to make it more effective.
• To be open and transparent. Openness about the effort’s process, successes, and problems can aid in convincing the community that its best interests are being served, thereby sustaining community support.

Who should be involved in creating formal public reporting processes?

Formal reporting is only useful if it informs community members about the effort. Therefore, those who would benefit from the effort and the reporting process, as well as those who can contribute, should be involved. These individuals will impart their current level of awareness about the effort, become more informed, and suggest their opinions in regard to reporting and transparency.

Some of the individuals you might involve include:

• Those who are most affected by the issue and would stand to benefit from a successful effort
• Local and other relevant officials
• The business community
• Community activists, particularly those who have been involved with the issue at hand
• Those who might see changes in their work if the effort gains momentum (for example, police might need further training in handling domestic violence)
• Those who will actually implement the effort

When should you create a formal public reporting process?

A formal reporting process should be created when an organization or effort begins implementation, so that the community understands its work and can see its progress over time. If, however, this has not happened, there are other times when creating a reporting process might be appropriate.

• When community support is needed. If community pressure, financial resources, or other community contributions are necessary for the continued functioning of the effort, it is important that the community has been informed of the benefits and rewards of investing in the effort.
• When there are questions about the effort. Issues may arise regarding the necessity of the effort, its cost, or other concerns. A reporting process can reveal the reality and invite the community into the discussion by presenting facts, rather than hearsay.
• When the issue the effort addresses has reached, or threatens to reach, dangerous proportions. This could be a special case of needing community support, or reporting may simply be needed to inform the community of the problem and how it will be remedied.
• To alert the community to one or more issues that have been recognized and are being addressed.

How do you create formal public reporting processes?

Assemble a group to plan the reporting process

The group should include representatives of all stakeholder groups as well as advocates. It might be helpful to also include members versed in communication, such as media professionals. The group should reflect the target audiences, and be familiar with their preferred ways of receiving information. The task of this planning group is to devise a reporting process that can reach, inform, and influence both key audiences and the general public.

Define the aim of the reporting process

In consultation with stakeholders, decide what you want to accomplish.

Some possibilities include:

• To comply with funders’ or government agencies’ reporting requirements
• To inform the public about the issue and your work
• To make sure that the effort gathers and analyzes the appropriate data so that the work, progress, and outcomes of the effort are accurately recorded and that you monitor progress and continually adjust its implementation
• To hold the effort accountable
• To gain public and/or policy support
• To raise funds

Consult with and collect information from stakeholders

Remember that stakeholders should include those who would benefit from and/or could contribute to the public reporting process.

There are a number of purposes for consulting with these individuals:

• To find out how much they know about the issue
• To find out what they know about the effort and/or the organization/ institution/coalition
• To find out the process and progress information that would be of interest
• To receive feedback about community needs and to inquire if there are modifications that need to be put into place
• To find out their opinions about what they see as the benefits of reporting, and what, if any, challenges they may see
• To enlist them, where appropriate, in becoming part of the reporting process
• To involve the community in a way that makes them feel ownership of the effort
• To be as fair, open, and accountable to the community as possible

To accomplish these purposes, you can conduct individual and group interviews, public forums, and informal meetings with key stakeholders and others.

Identify and prioritize the various audiences you want to reach

It is important to know who you are trying to reach, and what types of messages and information is most useful for each audience. Community audiences vary not only by their level of interest and their susceptibility to the issue, but also by their preferred channels of information, their understanding of and interest in different aspects of the issue and the effort, their level – or lack – of support, their political power, the language they speak, etc.

Some of the audiences that are likely to be targeted include:

• Populations affected by the issue
• Members of particular language groups
• Political officials and policymakers
• Others recognized as community leaders (CEOs of major businesses, heads of large institutions, community members respected for their integrity and/or influence)
• The general public
• Funders, public and private, which may include national or international foundations or development organizations

You may decide to prioritize your audiences depending on your goals for the reporting.

Understand the interests and preferred communication methods of the target population

Your reporting may need to convey a different message to each of these groups; for example, awareness of the issue for those affected by it, or the need for policy change for public officials. You may also need to frame and deliver the message differently for each population.

Another important consideration is the way in which different audiences prefer to receive information. Some may get most of their information from TV or radio, others from newspapers, others from the Web, still others from trusted community sources (clergy, health workers, teachers, etc.) In rural areas of the developing world, you might have to hold village meetings. You should deliver information to your audience through the appropriate medium of communication.

Timing is also important here. When can you attract the greatest attention and spread for your reports? Timing them to coincide with festivals, work holidays, special news broadcasts, ends or beginnings of legislative sessions or other specific periods might provide more exposure than end-of-the-quarter releases.

Tailor your reporting to each audience

Once you have decided on the priority audiences and have the information about their interests, preferences, and needs for receiving information, you can design messages that specifically target those audiences. This means couching the message in terms – and in a language – that members of that audience understand, using references to their own experience and concerns, and delivering it in a way that they are both likely to be exposed and in which they feel comfortable.

For example, a report on the progress of an immunization program that is aimed at Hispanic parents in a neighborhood of a large American city might be delivered in Spanish on a Spanish-language radio station. A report on a similar program in rural Nigeria might be delivered in the local language (even though English is the official language, most rural people do not converse in it) – at a village meeting or market.

Plan an ongoing communications campaign that includes annual reports

The reporting process should be part of an overall communication campaign that includes:

• Memorable messages about the issue and the effort
• Stories of people affected and their success in overcoming adversity
• Requests for help with the effort (advocacy, volunteers, cash and in-kind contributions)
• Regular periodic progress reports about the effort
• An annual comprehensive report on the state of the effort and the issue, with suggestions for adjustments to be made in the next year

All of these campaign elements should be communicated through a number of channels in a number of ways, in order to reach all the target audiences. They should employ as spokespersons trusted and respected figures – clergy, community leaders, media figures, athletes, etc.

Test your communications and public reports with the intended audiences

This is a three-step process:

•  Introduce them at focus groups and solicit feedback from target audiences
• Adjust the messages for greater effectiveness and accessibility based on feedback
• Adjust the overall communication plan, paying attention not only to feedback, but to cost as well

Assess and address public opinion about the effort

Based on your testing, determine what community members know and think about your effort. Do they have adequate and accurate information? Are they informed about the relevant issues, and whether or not they are affected by them? Do they see the value of the effort? Surveys, public forums, focus groups, and further interviews can expand on the perspectives you have received from piloting your messages.

Once you understand the state of public thinking about the effort, try to address the gaps in information and any negative feelings. Use public education programs, public meetings and forums, presentations to community groups, websites, and list servs to correct mistaken impressions, spread accurate information, and answer objections and any opposition arguments.

If any negative feelings, objections, or opposition arguments are legitimate – if there are gaps in the effort that could be bridged or aspects of implementation that need to be modified – you should acknowledge and remedy them. Nothing is gained by trying to defend elements of the effort that are in fact not well planned or not likely to work.
 
A particularly important strategy here is to develop a relationship with and use the media to help publicize information about the effort and requests to the community.

Educate public officials about the issues and the effort

Legislators and policy makers can be extremely helpful in publicizing and legitimizing the effort, and in helping to secure public support and funding. In addition, and more importantly, they can influence policy around the issue, leading to permanent positive change.

• Find a legislative or policy champion. The ideal champion for your effort is an elected official who is already sympathetic to your issue. If you can find a legislator who has had direct personal experience with the issue your effort addresses, it is likely that he or she will be willing to be a standard bearer in your campaign.
• Educate policy makers about the issue and your effort. Your champion can help by contacting friends and colleagues and adding legitimacy to your voice. A legislative insider can arrange for you to make a presentation to the legislative body, distribute your literature, and otherwise raise your profile among those who determine policy.
• Mobilize your local partners. Encourage those affected by the issue at hand and those with influence to contact legislators and policy makers. Elected officials, particularly, respond to the voice of the public; the louder that voice is – and the more influential those who are speaking – the more quickly officials respond.

Implement the communication and reporting campaign

Develop action plans for the campaign, dividing responsibility among members, and determining whom they will work with, what the timeline is, and what the results are intended to be. Then, use your tested communication and reporting format to report on the progress and results of the effort, explaining what the results mean and their implications for future work.

Encourage your intended audiences to use the reports to carry the effort further

Use the reports in sessions with key groups to reflect on and explain results, and to discuss what those results mean for further work. What adjustments do they imply, and why? Why do we think those adjustments will achieve particular results?

Use accessible language and explain everything in terms that are easily understood. Ask for feedback on how to improve the effort. And if you ask for feedback, be sure to take it into account. Either implement the suggested modifications – and communicate this to the community – or explain why you have chosen another route. Ultimately, the goal here is to use the information you obtain from the report and the feedback you receive from the community to make adjustments that render the effort more effective.

Continue the reporting and communications campaign

A public reporting process is not a one-time process. It has to continue over the long term in order to retain community involvement and support, both of which are necessary to the success of the effort.

In Summary

In the course of any community effort, it is important to establish and maintain communication with stakeholders and the community at large. This will include bidirectional information shared between members of the effort and the community.

Another aspect of communication , although informational as well, is more specific: reporting on the state and progress of the effort, on what has been learned from work so far, and on what kinds of changes need to be made in order to improve effectiveness. When community members are invited to give feedback which is then implemented in adjusting the effort, the community gains a sense of ownership and responsibility, leading to support for the effort. All of that, in turn, makes for a more successful effort and for a well-informed community that will be ready to tackle the next issue when it arises.

What do we mean by formal public reporting processes?

___It is conducted on a regular basis

___It is based on consistent measures

___It includes suggestions for adjustment and change in order to improve the effort

___It allows for feedback from its audience

___It continues throughout the life of the effort

Why create formal public reporting processes?

___To involve the community and garner support

___To bring the issue to the fore and raise the profile of the effort

___To hold the effort accountable

___To keep track of progress

___To collect new ideas for the effort

___To be open and transparent

Who should be involved in creating formal public reporting processes?

___Those who are most affected by the issue, and who would therefore stand to benefit from a successful effort

___Local and other relevant officials

___The business community

___Community activists, particularly those who have been involved with the issue at hand

___Those who might see changes in their work if the effort gains momentum

___Those who will actually implement the effort

When should you create formal public reporting processes?

___When community support is needed

___When there are questions about the effort

___When the issue that the effort addresses has reached, or threatens to reach, dangerous proportions

___To alert the community to one or more issues that need addressing, and to the fact that something is being done about them

How do you create formal public reporting processes?

___Assemble a representative group to plan the reporting process

___Define the aim of the reporting process

___Consult with and collect information from stakeholders and others who would benefit from and/or could contribute to the public reporting process

___Identify and prioritize the various audiences you want to reach, and what types of messages and information each might receive

___Understand the interests and preferred communication methods of those the report is aimed at

___Tailor your reporting to each audience you want to reach

___Plan an ongoing communications campaign that includes annual reports

___Test your communications and public reports with their intended audiences during development

___Assess and address public opinion about the effort

___Educate public officials about the issues and the effort

___Implement the communication and reporting campaign

___Encourage your intended audiences to use the reports to carry the effort further

___Continue the reporting and communications campaign over the life of the effort