IX. MONITORING, EVALUATION AND IMPACT ASSESSMENT:
DEVELOPMENT AND UTILIZATION OF UNFPA INDICATORS*

Technical and Policy Division, UNFPA

Despite the economic reforms and restructuring undertaken by many developing countries, it has become increasingly apparent that the achievement of sustained economic growth and sustainable development, and particularly the eradication of poverty where it is widespread, cannot be taken for granted. Donors and recipients of technical assistance alike have become increasingly aware of the need for sound statistical indicators to track progress towards the achievement of important development goals. Not surprisingly, therefore, following the International Conference on Population and Development in 1994, much greater emphasis has been placed on the need to measure progress in achieving population and reproductive health goals. The use of indicators to serve as markers of programme performance has thus assumed new prominence.

UNFPA has developed a comprehensive set of indicators for population and reproductive health programmes, covering three principle thematic programme areas, reproductive health, population and development strategies and advocacy and information, education and communication (IEC) which are taken as one thematic area, and for use in conjunction with the logframe.The present paper discusses the framework of the UNFPA indicators, their selection criteria; decision-making on data collection for the indicators; strengths and weaknesses in utilizing existing data sources; and the implications for data collection of compiling the indicators.

Background

Since the 1994 Conference, a number of United Nations-sponsored global conferences and summits have been convened, including the World Summit for Social Development, held at Copenhagen in 1995, the Fourth World Conference on Women, held at Beijing in 1995, the Second United Nations Conference on Human Settlements, held at Istanbul in 1996, and the World Food Summit, held in Rome in 1996. Each of these conferences and summits, like those held in the early 1990s, set out explicit goals and programmes of action with implications for the development and use of appropriate indicators. In response, four key sets of indicators have been developed for use in monitoring global conference goals:

A core set of 36 global indicators for use in common country assessments which are held as a basis for developing the United Nations Development Assistance Framework (UNDAF) in countries
The Minimum National Social Data Set (MNSDS) of 15 social sector indicators which was endorsed by the United Nations session held in 1997
The Administrative Coordinating Committee Task Force on Basic Social Services for All promulgated the use of 11 key social service sector indicators based on the goals of the global conferences
The set of 31 indicators proposed by the Development Assistance Committee of the Organisation for Economic Cooperation and Development for the measurement of progress towards the achievement of the development goals set out in Shaping the 21st Century:The Contribution of Development Cooperation1. An important addition in this list is the inclusion of indicators to track gender equality
Each of the above-mentioned lists is made up of outcome or impact indicators together with a few reference indicators, and several indicators are common to all lists. The distinctiveness of the UNFPA set of indicators is its central focus on process indicators, in addition to outcome indicators, for use in the measurement of performance of population and reproductive health programmes. It follows that the UNFPA set is both more narrowly focused and contains a much larger number of indicators than in the four global sets mentioned above.

A. The framework and its links to the logframe

The framework for the UNFPA set of population and reproductive health indicators is built on two related structures. The first structure provides a stratification of the indicators according to the three thematic areas used by UNFPA in it programming work. Accordingly, separate sets of indicators are proposed for reproductive health, population and development strategies, and population advocacy and IEC, which are here taken together as one thematic area. Special attention is given to indicators that address gender issues in all thematic areas. Within each of the three areas, the indicators are further divided into sub-thematic component groups.

The second structure is designed for use in conjunction with the logframe, which is a methodology that can be used for preparing country programmes, subprogrammes and component projects. Indicators are integral to this methodology and serve as markers to measure programme performance. The internal logic of the logframe approach, whereby the achievement of aims, or interventions, is measured against objectively verifiable indicators, provides an important potential use for the UNFPA indicators. Aims are measured at four levels, activity, output, purpose and goal. Activities and outputs are generally programme or project deliverables, whereas purposes and goals reflect changes in behaviour and impact.

Thus, within the logframe, separate indicators are used for measuring changes at different levels. The outcome or impact indicators measure changes in bahaviour, while the process and management indicators measure programme performance. Traditional indicators are perceived to focus on outcomes, to operate at the global or national levels, and to be based on official statistical sources. The shift to a more programmatic approach, therefore, provides challenges in both the construction of appropriate indicators and identifying reliable sources of information. Central to this shift is the notion that improved monitoring and evaluation of country programmes is essential for more effective service delivery. Linkages between the various types of indicators and the logframe axes are presented schematically in the table below.

The cross-classification of the above-mentioned structures, that is, stratification in terms of thematic area and the logframe, reflects the way in which the indicators can be used. As the main platform of UNFPA programmes is reproductive health, indicators for this sector will reflect a concern that programmes contribute to achieving the global goals of the International Conference on Population and Development. While some attempt is made to provide new process indicators to support programme managers, many of the indicators in this thematic area are focused on outcomes or impact. This is also the case for those indicators provided for the area of population and development strategies.

In contrast, IEC and advocacy are seen primarily as operating in support of reproductive health programmes. The desired outcomes for this thematic area are inextricably linked to the idea of changed reproductive behaviour through the utilization of quality reproductive health information and services. This supportive role of IEC and advocacy is reflected in the nature of the indicators proposed, emphasizing process rather than outcome.

B. Key indicators

Within the above-mentioned framework, UNFPA developed the set of just over 100 indicators. The determination of a short sub-set of key indicators for use in population and reproductive health programmes will depend on a number of factors, including the nature of the problem being addressed in relation to the local setting, as well as the availability and cost of the data required to construct them. The annex table gives a short list of 20 key thematic indicators that may provide summary baseline information for assessing the performance of population and reproductive health programmes. The indicators on this short list will often suffice where choices have to be made. The maternal mortality ratio and the percentage of the population with access to primary health care services would be useful additions to this list. However, in practice, that ratio is extremely difficult to measure in most countries, primarily because maternal deaths are relatively infrequent events. Moreover, the definition of access can be extremely problematical and an adequate study of accessibility of services will often require focus on a multiple set of indicators.

C. Criteria for indicator selection

The selection of each indicator raises technical questions about the implications for data collection and processing, as well as other operational and financial issues. Assuming that indicators can be developed from existing data sources, it is essential that those selected be relevant and robust for the purposes that they will serve. For some of the indicators, the basic statistical sources required to construct them already exist and the major tasks are to ensure their clear definition, consistent use and proper interpretation. For example, indicators to track the pursuit of universal access to primary education and closure of the gender gap by 2005 are available in most countries, even though they might not have been utilized in the systematic way now proposed. For other goals of the Conference, however, considerable innovative thinking is required to develop appropriate indicators. This is the case, for example, with respect to defining or conceptualizing indicators to measure progress towards achieving the full integration of family planning and reproductive health services, and measuring the quality of reproductive health services.

The process of selecting the comprehensive set of UNFPA indicators has involved several meetings and consultations among technical experts within and outside UNFPA. While it is difficult to specify a unique set of criteria that has survived in the selection of the indicators, it is possible to summarize some of the more important technical issues that have arisen and the guidelines that have been followed.

Indicators should not impose an unnecessary burden on reporting agencies. As far as possible, therefore, where the needs of the various users are similar, the demands for indicators should be harmonized
As far as possible, indicators should be Where source materials either do not exist or are too weak to warrant inclusion, programme efforts to strengthen existing sources or create new ones able to provide accurate and timely data must be feasible
Indicators should provide robust measures of progress towards the goals of the Conference, and for monitoring performance in delivering programme outputs at the various levels of the logframe
Indicators should be quantifiable and capable of consistency of measurement, that is, repeatable. At the minimum, this has been interpreted as providing two mutually exclusive response categories, but more usually refers to the ease of representation of an indicator as a frequency, percentage, ratio and so on
The purpose of each indicator should be clear for programme managers to understand, and the indicators should be relatively straightforward to interpret: of course, it is recognized that any single indicator will always have some limitations

D. Decision-making on data collection

The costs and benefits of developing and using indicators must be carefully weighed. The collection and construction of accurate, relevant and up-to-date indicators will clearly have a cost component, but the indicators will permit pertinent questions about the progress of the programme or project to be posed and answered. Answers to such questions, among others, must be seen as prerequisites to making decisions about the appropriateness of strategies to be pursued, and will provide a basis for decisions to be made regarding the commitment of additional financial and human resources to the programme or project.

For this reason, it is important to establish procedures to be followed in situations where the statistical inputs for the indicators are not routinely available. In practice, countries will need to focus on strengthening non-traditional sources of statistical information. A considerable amount of potentially useful data already exists-clinic records, service statistics, hospital records, training records, general health statistics systems, and government records, as well as record-keeping systems established by the private sector and NGOs. These largely untapped data from various sources, with some skilled inputs and financial support, could provide a range of valuable data across a broad spectrum of activities. It is essential, therefore, that a review of these sources be undertaken as a preliminary step in defining indicators appropriate for a particular programme or project, and in setting up procedures for collecting, processing or assembling the relevant data.

E. Sources of data

1. Housing and population censuses

These provide excellent sources for many of the broad macro-level indicators required to measure progress in the area of population and development. The population censuses include such indicators as measures of population size, structure and distribution; school attendance and education levels attained; employment (by occupation and industry), unemployed and the non-economically active population; and measures to track gender equity and equality. The housing censuses include questions on materials used for housing; persons per room; and access to potable water, electricity and sanitation. The universal coverage provided by a national census is of great benefit in generating indicators or tables relating to specific geographic areas or subgroups of the population.

However, censuses have two major limitations in their use for programme or project monitoring. First, the scope of a census is usually very restricted and does not include specific questions that relate closely to outputs of activities implemented in the UNFPA thematic areas. Second, the frequency of producing a population census is usually not sufficient to provide the regular feedback of performance evaluation essential to effective management.

2. Sample surveys

Sample surveys provide a rich source for data relating to reproductive health knowledge, practice and behaviour. Survey questions and probes tend to be more detailed than censuses. Moreover, because of their smaller scale, surveys tend to employ better qualified and trained enumerators than censuses do, and as a result the data are more reliable. However, the shift to a more programmatic approach to the construction of indicators sets limits on the usefulness of large sample surveys, with their emphasis on measuring economic, social and demographic behaviour. First, the limited coverage in selected project areas means that they cannot be used to provide the intensive reports of activities required for project monitoring. Second, large sample surveys in most countries are conducted too infrequently to provide the regular information required for project monitoring. Third, the agencies responsible for successive surveys can change the scope, coverage, design, definitions and field procedures. They can introduce problems in comparability over time as well as the conduct of surveys at irregular intervals. Fourth, most demographic and health surveys have not covered the reproductive health needs of adolescents and men, married and unmarried.

There have been some promising initiatives in the use of small tailor-made surveys, for example with coverage restricted to project areas or exit interviews at supply distribution points. The results of focus group discussions and records maintained by specialized service providers can provide ideas to help in refining indicators and strengthening data collection partnerships.

3. Civil registration

The civil registration system is potentially the best source for many of the indicators, including birth rates, death rates, infant mortality rates, and maternal mortality ratios. Its great strengths lie in the detail it collects-on the newborn, on the parents, or on the deceased-and the authority of the reporting. An added benefit is the continuous nature of reporting that permits rates to be calculated for any defined reference period. Unfortunately, in many countries vital registration is far from complete. Rural notification of births and deaths is particularly poor, with the result that data from hospitals over-represent urban dwellers and the sick. Poor and fluctuating coverage also makes it difficult to construct time series analyses.

4. Other formal data collected by the national statistical office

National statistical offices provide rich sources of data for the construction of indicators. Very often, the published data are inadequate to meet the needs of the project, yet unit-level record files exist and tabulations can be generated readily to meet the programme managers' needs. Other statistics generated by the national statistical office relating to the project areas can also prove useful, and can probably be accessed at a low cost to the project. These could also form the basis of performance indicators. Regular household income and expenditure surveys, poverty or nutrition surveys, especially if they cover the project areas, can prove very useful. More general statistics, such as estimates of gross domestic product, balance of payments or trade, can be assembled by the project manager to provide a statistical profile of the country and to track changes over time.

5. Health information

Moving closer to the specific needs of the project manager to provide indicators relating to the reproductive health programme, either nationally or in selected project areas, a careful review is needed of the status of health statistics, especially in the health department, which potentially provides a most valuable source for reproductive health data. Several elements of the health information system may exist, with data being collected by different units or agencies. Some are likely to be more complete and reliable than others.

Typical problems related to health information exist, such as incomplete clinic or patient records, lack of consolidated reporting to the health information unit at the centre, varying quality of data across regions, and shortage of expert assistance to analyse and interpret findings. Despite these difficulties, a sustained effort could be made to resolve problems in the UNFPA project areas with the full cooperation of the health department.

6. Other administrative records

Record-keeping and management information systems maintained by government agencies, NGOs and other branches of civil society provide a wide range of inputs that could be tapped for the construction of indicators. It is essential, therefore, that a review of these systems be undertaken as a preliminary step in defining indicators appropriate for a particular programme or project and setting up procedures for collecting, processing and assembling relevant data. It is recommended that the project director should establish links with the planning office or some other authoritative agency to serve as a reliable respondent on the status of any initiatives.

7. Project management statistics

Responsibility for maintaining reliable sources for the construction of the indicators lies with the project manager. In the design stage of the project, it will be necessary to undertake a review of existing sources of data and their deficiencies. This review will influence the choice of indicators selected and the commitment to assemble data. Although many sources may be of potential value, they are likely to need strengthening to provide the needed indicators for UNFPA project management. Where cooperation with government or NGOs is possible, a clear programme will be needed, setting out the steps to be taken to improve the statistical infrastructure of the project areas.

F. Implications for data collection

The proposals for the construction and use of a set of performance indicators in priority countries must include the provision of resources for strengthening the sources of data. Without this provision, many of the indicators will not be sufficiently robust or comparable over time to serve the purposes for which they are intended. Further, successful programme or project management involves the establishment of appropriate information systems. Such systems can draw upon existing data, but managers also need to recognize the limitations in the range and quality of existing sources and ensure that additional information is collected within the framework of the programme or project to permit effective and regular monitoring.

Going beyond programme management, integrated national statistical systems can provide access to a range of national data required to produce key indicators and meet a wide range of user needs. To ensure sustainability, bold initiatives will be required to strengthen data collection, storage, retrieval and management and to develop user-friendly databases. These are monumental tasks that require coordinated efforts from the various government agencies, NGOs and civil society, and the donor community. It is likely that, through the preparation of this set of indicators, awareness can be enhanced about existing shortcomings in collecting and utilizing data, and a more open dialogue can be promoted between the various information gatherers and potential users.

The move towards constructing a common set of indicators for each country has obvious benefits. First and foremost, it enables aggregation of the indicators throughout the various levels, country, region and world. More than this, however, the discipline inherent in the pursuit of common indicators provides momentum to countries in achieving uniform and rigorous technical standards. This process is particularly important in the building of common frameworks, the formulation of standard concepts and definitions, and the development of standard procedures and methodologies. The idea underlying standardization is also important in considering the cost of data collection, particularly for expensive sample surveys. Once an indicator has become accepted globally, possibly meeting the needs of a number of users from within the country and representing different agencies, it is likely that the inclusion of specific questions to provide data for the construction of the indicator will become a matter of course. These questions would then presumably be included automatically in the regular survey programmes. This process of globalization is characteristic of the way survey questions are being adopted, illustrated by the near universal use of indirect questions to measure fertility and mortality. It does not follow from the above discussion that UNFPA is asking for and prepared to finance additional surveys.

G. Conclusion

Indicator selection is a dynamic and complex process and any menu of indicators will necessarily change. Those indicators that serve their purposes will be retained, those that fail will be eliminated, while new ones will be added in response to changing needs. An important consideration in this refinement process is how to balance the needs for comprehensiveness, consistency and relevance with some of the technical and cost constraints. The UNFPA new process indicators have not yet been thoroughly field-tested. This difficult and expensive but invaluable exercise can only be implemented through systematic efforts, nationally, regionally and globally. A key challenge is to collect and use contextually relevant programme indicators to help bring about more effective performance in population and reproductive health programmes.

End Notes

* This note is a summary of the comprehensive report prepared by UNFPA entitled Indicators for Population and Reproductive Health Programmes (forthcoming).

1 Organization for Economic Cooperation and Development (Paris, 1996).