National Initiatives

Performance Measurement and Evaluation
Resource Handbook

January 2004

Introduction

What is the Purpose of the Resource Handbook?

To assist service providers, provincial representatives, CIC officers, and other stakeholders in adopting a common vocabulary when discussing settlement program results and accountability.

The Handbook provides definitions of commonly used Results-Based Management terminology as well as draft program-specific Logic Models. These definitions are based upon those developed by the Treasury Board Secretariat which can be found online at http://www.tbs-sct.gc.ca/eval/tools-outils_e.asp

Results

Why are Results Important?

Over the past ten to fifteen years, there has been increasing pressure on governments to focus on results.

Citizens have clearly expressed that they expect to receive good value for their tax dollars and services that meet their priorities. Governments have recognized the importance of demonstrating the effective and efficient use of public resources, applying sound risk-management practices, and reporting on results.

Results for Canadians

Canada is no exception and the management framework for the federal government, Results for Canadians, stresses the importance of looking beyond activities to focus on results and benefits to Canadians.

Results for Canadians also notes that measuring and evaluating results allows organizations to reward success, learn from experience, and build public confidence.

Results for Canadians can be found on the Treasury Board Secretariat website at http://www.tbs-sct.gc.ca/res_can/rc_e.asp

Treasury Board Policy

The Treasury Board Policy on Transfer Payments also emphasizes results and accountability. The Policy states that departments with transfer payment programs must seek Treasury Board approval before March 31, 2005 to replace or renew terms and conditions pertaining to these programs.

The Policy sets out clear requirements for departments to establish a Results-Based Management and Accountability Framework, which provides for appropriate measuring and reporting of results. The Policy can be accessed at http://www.tbs-sct.gc.ca/pubs_pol/dcgpubs/ TBM_142/ptp_e.asp

Results-Based Management and Accountability

What is Results-Based Management?

Results-Based Management is a participatory and team-based approach to management that seeks to focus on an organization's or program's efforts on the achievement of results.

Results-Based Management involves defining realistic expected results, assessing risk, monitoring progress towards the achievement of results (performance measurement), integrating lessons learned into management decisions, and reporting them in simple, understandable ways.

What is Accountability?

Accountability refers to the responsibility of managers of all publicly funded programs to account for how funds are being spent and what benefits accrue to participants and society as a result of these initiatives.

Contribution Accountability Framework

What is the Contribution Accountability Framework?

In 1999, CIC launched the development of an accountability framework for its settlement and resettlement contribution programs.

This Framework will ensure the accountability of settlement expenditures through monitoring of service delivery and evaluation of program effectiveness. Information collected will also be used to identify program changes that will help enhance the capacity of newcomers to integrate into Canadian society.

Finally, implementation of the Framework will also contribute to the Department's ability to meet the Treasury Board Policy on Transfer Payments.

The Framework covers four settlement contribution programs:

• Language Instruction for Newcomers to Canada (LINC);
• Immigrant Settlement and Adaptation Program (ISAP);
• Host Program; and
• Resettlement Assistance Program (RAP).

There are five key elements in the Contribution Accountability Framework (CAF):

• performance measurement;
• evaluation;
• contribution agreement process;
• management control (where applicable, under an agreement); and
• Provincial-Territorial accountability.

Accountability Tools

This Handbook will focus on performance measurement and evaluation as two key tools available for accountability:

Performance measurement (program monitoring) is the ON-GOING collection of, and reporting on, information on program implementation to guide corporate decision-making. This information will be collected largely through iCAMS, as described on the following pages.

Evaluation is the PERIODIC systematic collection and analysis of information on the performance of a policy, program or initiative to make judgements about relevance, progress or success and cost-effectiveness and/or to inform future programming decisions about design and implementation. CIC has developed an evaluation strategy for its settlement programs. The strategy can be accessed at http://integration-net.ca/english/ini/caf-cipc/index.htm

Performance Measurement

What is iCAMS?

The Immigration-Contribution Accountability Measurement System (iCAMS) is an Internet-based database that has been developed for service providers to report information about their clients and services to CIC. iCAMS is an integral part of CAF's performance measurement component.

What information is being collected?

CIC will use iCAMS to collect quantitative performance measurement information. The following are examples of the kind of client information that is collected by iCAMS:

• IMM1000 (record of landing) or other immigration identification client number
• name
• address
• date of birth
• gender
• country of birth, citizenship and last permanent residence
• educational and language background
• marital and family status
• date of arrival and landing

Besides client information, iCAMS will also collect information on the services and programs that service providers deliver to clients.

What is the Purpose of iCAMS Data Collection?

The information collected from service providers will give CIC a more reliable picture of the concrete RESULTS of each program. In particular, iCAMS data will be used for the following purposes:

• program monitoring (performance measurement)
• program evaluation
• policy analysis
• research

Evaluation

Program evaluations provide additional quantitative and qualitative data. Evaluations of settlement programs are envisioned as part of the Contribution Accountability Framework implementation. Evaluation will begin with the Language Instruction for Newcomers to Canada (LINC) program in 2003/04 and continue with the Immigrant Settlement and Adaptation Program (ISAP) and Host program in 2004/05.

Evaluations will include wide-reaching, participative discussion on program outcomes. As part of pre-evaluation activities for its settlement and resettlement programming, CIC will be hiring an external Results-based Management Consultant to conduct RBM workshops with service providers across the country. These workshops will aim to obtain feedback on the draft results statements and indicators for each of CIC's settlement and resettlement programs. Draft Logic Models for these programs can be found in the pages that follow. RBM workshops for the LINC program were held in November of 2003. ISAP and Host RBM workshops will be held in February and March of 2004.

It is important for programs to clearly articulate the results they aim to achieve, and the indicators that will be used to measure those results, prior to evaluation. Programs need to know what they want to achieve in order to later determine whether or not the desired results are being achieved.

Key Results Terminology

Outcomes (results) are changes that occur as a result of program activities and outputs, and can be divided into immediate outcomes (short term), intermediate outcomes (medium term) or final outcomes (long term).

Results Chain

To put the table shown in the previous page in a more visual way…
the Results Chain…

The Results Chain shows the causal or logical relationship between inputs, activities, outputs and results. This chain can also be seen in a program Logic Model.

Measuring Results

What are Indicators?

It is not enough to define results. Results must be measured. Indicators are used to assess whether or not results are being achieved.

Indicators can be qualitative or quantitative. Qualitative indicators are judgements and perceptions. Quantitative indicators are measures of quantity.

Indicators can be pointers, measures, numbers, facts, opinions, or perceptions that help measure progress towards change. Examples of indicators include:

Examples of Indicators

• Increase in…
• Decrease in….
• Number of…
• Percentage of …

Measuring Device

• Pointer
• Number
• Fact
• Opinion
• Perception

Program-Specific Logic Models

Please see Annex A for the final LINC logic model. Consultation on this model took place in November of 2003.

In Annex A, you will also find draft ISAP and Host logic models, that will be used as the basis for the workshops on results.

Data Elements

In order to build indicators, data must be collected.

Data elements are the actual pieces of information that are being collected, such as the number of clients using services, by type.

Data will be collected from service providers through iCAMS and through evaluation.

CIC has agreed to collect individual data for LINC and RAP. This collection should not present a problem, as LINC is a standardized, non-sensitive service. RAP is even more standardized, and the nature of RAP demands that client information be provided to CIC for income support purposes.

For the ISAP and Host, CIC has agreed that aggregate client service usage data can be provided, along with a separate list of client information, including identifier numbers. Data collection for ISAP and Host on an individual basis by service providers will be voluntary.

Importance of Baseline Data

Baseline data is particularly important. It is difficult, if not impossible, to measure change over time without baseline data. Baseline data provides an established line of information from which to measure progress. In iCAMS, it allows historic data entered for clients to be accessible.

The data elements on the next few pages will be the foundation for reports produced with iCAMS. iCAMS will be able to run reports based on these data elements and client profile characteristics. The report designs will be affected depending on whether service information is provided on an individual or aggregate basis.

In order to ensure the security and privacy of information collected through iCAMS, CIC has implemented recommendations made by the Privacy Commissioner of Canada; hired an independent third party to conduct a Privacy Impact Assessment; and developed information and training tools, such as a privacy pamphlet and poster informing clients of the purpose of information collection in iCAMS, as well as a privacy manual for CIC staff. For more information on Privacy and Security please visit the Contribution Accountability Framework website at http://integration-net.ca/ /english/ini/caf-cipc/index.htm

Profile characteristics of clients

• age
• gender
• country of birth
• country of citizenship
• country of last permanent residence
• destination city
• mother tongue
• immigration category
• special program code
• educational qualifications
• years of schooling
• official language ability
• marital status
• family status
• date of landing
• Canadian citizenship
• visa office

These lists of data elements and definitions were produced as a result of discussions with the Performance Measurement Advisory Committee, NHQ National Program Working Groups, Settlement Management, and the Contribution Accountability Team.

Data Elements - All Programs

The following information will be collected for Language Instruction for Newcomers to Canada (LINC), Immigrant Settlement and Adaptation Program (ISAP), Host Program, and Resettlement Assistance Program (RAP).

ALL PROGRAMS

1. FOSS Client ID #, IMM 5292/5509/1000#, OR Temporary Resident/Minister's Permit Number
2. CIC Given Name and Surname
3. Date of Birth
4. Home address, city, province, postal code

Specific data elements for each program can be found in the iCAMS training manuals.

Annex A:

• LINC/CLIC Logic Model
  >> Adobe pdf size: 42 K 1 p
• ISAP Logic Model
  >> Adobe pdf, size: 28 K 1 p
• Host Logic Model
  >> Adobe pdf, size 27 K 1 p

Back to top
Settlement Programs Evaluation Strategy