National Initiatives

>> Adobe® Acrobat format, 847 K 60 pp

Prepared for:
Citizenship and Immigration Canada

Prepared by:

Dr. Gillian Kerr

RealWorld Systems
www.realworldsystems.net

May 3, 2004

Acknowledgements

This Settlement Evaluation Framework is the product of several research activities. The current version of the document was prepared by RealWorld Systems, based closely on a draft Settlement Evaluation Framework that was prepared by Goss Gilroy Inc. (GGI) and submitted to CIC in January 2004. Most of Section 2 and large parts of other sections, along with the evaluation framework and logic model for LINC, were retained from the GGI version with little or no alteration. The revised sections incorporate findings from a series of 14 workshops across the country carried out by RealWorld Systems with representatives of service provider organizations delivering the Immigrant Settlement and Adaptation Program (ISAP) and the Host Program. RealWorld Systems also consulted recent literature on settlement research as well as several key informants, including CIC representatives.

The Settlement Evaluation Framework prepared by GGI in January 2004 was itself based on a number of activities. Firstly, GGI carried out a series of workshops with representatives of service provider organizations delivering the Language Instruction for Newcomers to Canada (LINC) Program. Secondly, GGI interviewed several LINC, ISAP and Host program officials at CIC Headquarters in Ottawa to update the descriptive program information for those programs. Thirdly, the Framework drew on information in the Evaluation Frameworks prepared by GGI for each of the settlement programs (LINC, ISAP and Host) in 1999 and 2000.

Goss Gilroy Inc. and RealWorld Systems would like to thank all the individuals who participated in these projects for their time and valuable contributions. As well, the assistance of CIC Headquarter and Regional representatives, who took the time to review the draft deliverables and provide suggestions for improvement, was greatly appreciated.

Table of Contents

1. INTRODUCTION

2. BACKGROUND

2.1 HISTORY OF SETTLEMENT PROGRAMS

3. DESCRIPTION OF CIC PROGRAMS

3.1. PROGRAM RESOURCES

3.2. ROLES AND RESPONSIBILITIES

3.2.1. Settlement Division

3.2.2. CIC Regional Offices

3.2.3. CIC Local Offices

3.2.4. Service Providers

3.3. PROGRAM DESCRIPTION

3.3.1. Assessment of Needs and Selection of Service Providers

3.3.2. Support of Service Providers

3.3.3. Promotion of Settlement Programs

3.3.4. Language Instruction for Newcomers to Canada (LINC)

3.3.5. Immigrant Settlement and Adaptation Program (ISAP)

3.3.6. Host Program

4. EVALUATION QUESTIONS, OUTCOMES, INDICATORS AND OUTPUTS

4.1. METHODS AND DATA SOURCES

4.2. ATTRIBUTION PROBLEMS

4.3. COMMON QUESTIONS FOR ALL SETTLEMENT PROGRAMS

4.4. QUESTIONS FOR SPECIFIC SETTLEMENT PROGRAMS

4.4.1. Language Instruction for Newcomers to Canada (LINC)

4.4.2. Immigration and Settlement Adaptation Program (ISAP)

4.4.3. Host Program

5. SETTLEMENT PROGRAM LOGIC MODELS

Settlement Programs Logic Model

LINC Logic Model

ISAP Logic Model

Host Logic Model

1. INTRODUCTION

Citizenship and Immigration Canada (CIC) is in the process of evaluating all of its national settlement programs. This report lays out an evaluation framework and logic models for CIC's programs, and defines some key questions that will guide the program evaluations.

The framework aims to clarify the desired results of CIC's settlement programs for their many stakeholders – CIC staff, Service Provider Organizations (service providers), evaluators, and senior members of the federal government. Like all frameworks, it should be revised and updated as new research expands current knowledge of what interventions lead to successful settlement and integration.

Stakeholder groups will use this framework in different ways:

CIC staff at National Headquarters will use the framework to define the requirements for evaluations for the settlement programs in 2004. In addition, the output variables and activities contained in the framework may lead to revisions of some iCAMS variables to reflect changes in activities and recommended outputs.

CIC staff in the Regions can use the framework to guide contribution negotiations with service providers. For example, service providers who can demonstrate success in key immediate outcomes may be able to make a case for increased funding by meeting CIC objectives for greater service effectiveness.

Service Providers can use the framework to help them develop data collection processes that will allow them to demonstrate success in meeting program outcomes. Service providers can also use the program logic models to review their service delivery models so that their activities lead logically to desired immediate outcomes.

The first draft of the evaluation framework for CIC's settlement programs was written in 1999/2000. Since then the framework has been significantly revised, based on dozens of meetings, workshops and conversations with service providers and CIC staff. Interviews with key informants and a review of the literature on settlement and labour market integration also contributed to the revisions. The evaluation framework for LINC was discussed in workshops with service providers in the fall of 2003, and a similar exercise (involving 14 workshops across the country) was carried out for the ISAP and Host programs in February and March 2004. The program logic models in Section 5 distill hundreds of comments and suggestions into four highly simplified diagrams.

Following are some key issues that should be kept in mind while reviewing this document.

• It is long past time for the social services to evaluate themselves critically, and the settlement sector is no exception. The transition towards accountability and results-based management, while challenging, is essential for the achievement of Canadian policy objectives as well as the individual goals of newcomers themselves. Service providers across the country acknowledge the importance of this process, and generally welcome the opportunity to demonstrate and improve their effectiveness.
• On the other hand, data collection is time-consuming for service providers, and should be minimized as much as possible while meeting accountability requirements and evaluation objectives. A common mistake for both funders and service providers is to gather data elements that don't lead directly to better services or improved impact, but instead reduce efficiency by taking up scarce time. This evaluation framework attempts to minimize data collection to measures that will have a real benefit on improving services to clients and enabling CIC to manage agency performance.
• In particular, indicators of intermediate and long term outcomes are expensive to collect and difficult to validate. Over the next few years, starting with the upcoming evaluations, both CIC and service providers should develop and begin collecting meaningful and valid indicators to create a baseline for evaluating settlement programs in the future. CIC should identify service providers and researchers that have demonstrated leadership in this area, and work with them to enhance the capacity of the settlement sector to evaluate itself.

Section 5 of this paper – Settlement Program Logic Models – is intended to be used as a stand-alone summary of the evaluation framework. It contains the logic models for all three programs, the logic model for CIC's settlement programs as a whole, and a few notes on implementation.

<< contents | next >>