Settlement Programs
Evaluation Strategy
June 2003
TABLE OF CONTENTS
INTRODUCTION
GOVERNMENT OF CANADA ACCOUNTABILITY
CIC CONTRIBUTION ACCOUNTABILITY FRAMEWORK (CAF)
Performance Measurement
Evaluation
Contribution Agreement Process
Management Control Framework
Provincial-Territorial Accountability
DEVELOPING A CAF EVALUATION STRATEGY
RELATED ASSUMPTIONS, ISSUES, AND FACTORS
Work Plan/Timelines
Cost
Program Stakeholder Roles and Responsibilities
ATTACHMENTS
Attachment
A
Attachment B
Attachment C
INTRODUCTION
Newcomer participation in Canadian society is in Canada's social and
economic interest. To encourage this participation, Citizenship and Immigration
Canada (CIC) provides funding to a variety of organizations that deliver
settlement services. Settlement services are supported through three CIC
settlement programs that assist immigrants and refugees in learning about
life in Canada, accessing public services, acquiring basic language skills
in one of the official languages, and becoming participating and contributing
members of society:
- Language Instruction for Newcomers to Canada Program (LINC)
- Immigrant Settlement and Adaptation Program (ISAP)
- Host Program (HOST)
CIC also supports resettlement through the Resettlement Assistance Program
(RAP).
CIC settlement programs are implemented in all provinces and territories
in Canada with the exception of Manitoba, Quebec, and British Columbia.
These three provinces receive transfer payments from CIC to manage and
deliver comparable settlement programming. In 2001/02, federal spending
on settlement programming was approximately $333 million.
GOVERNMENT OF CANADA ACCOUNTABILITY
Results for Canadians
As with all publicly-funded programs, there is an obligation to account
for how funds are spent and what benefits accrue to participants and to
Canadian society as a result of program initiatives. On March 30, 2000,
the Treasury Board of Canada published Results for Canadians: A Management
Framework for the Government of Canada. Results for Canadians stresses
the importance of looking beyond activities to focus on results and benefits
to Canadians. It also notes that measuring and evaluating results allows
organizations to reward success, learn from experience, and build public
confidence.
Treasury Board Policy on Transfer Payments
All government departments with grants and contributions programs are
also required to meet the requirements of the Treasury Board (TB) Policy
on Transfer Payments, effective June 1, 2000. The TB Policy states that
departments with transfer payment programs must seek TB approval before
March 31, 2005 to replace or renew terms and conditions pertaining to
these programs. The Policy also requires departments establish and use
a Results-based Management and Accountability Framework (RMAF).
The RMAF includes the following:
- Description of the roles and responsibilities of the main partners
involved in delivery of the program;
- Results-based logic model showing activities, outputs and a chain
of outcomes for the program;
- Performance measurement strategy demonstrating how progress will be
tracked, what indicators will be used to measure progress on an ongoing
basis and to make adjustments and improvements;
- Description of planned evaluation activities, including what will
be evaluated, how, when, and if possible, at what cost; and
- How reporting will occur, what it will include, who will provide it,
and to whom it will be addressed.
CIC will adopt results-based management practices as a requisite for
meaningful program evaluation. CIC will therefore undertake an exercise
to define program-level outcomes and indicators before proceeding with
formal evaluations of its settlement programs.
2001 Report of the Auditor General
The 2001 Report of the Auditor General on Voted Grants and Contributions
encouraged CIC to continue its efforts to measure and report on the results
of its settlement programming.
CIC CONTRIBUTION ACCOUNTABILITY FRAMEWORK
In 1999, CIC launched the development of an accountability framework
for its settlement and resettlement contribution programs. The CIC Contribution
Accountability Framework (CAF) is meant to ensure the accountability of
departmental expenditures on settlement and resettlement programs, monitor
service delivery, and evaluate the effectiveness of these programs in
meeting the needs of newcomers. Information collected will also be used
to identify program changes that will help enhance the capacity of newcomers
to integrate into Canadian society.
There are five CAF components:
- Performance Measurement
- Evaluation
- Contribution Agreement Process
- Management Control Framework
- Provincial-Territorial Accountability
Priorities in the last fiscal year (2002/03) were focused on performance
measurement, the contribution agreement process, and provincial/territorial
accountability, as well as increased improved communications with stakeholders.
Performance Measurement
The Immigration-Contribution Accountability Measurement System (iCAMS),
used by service providers to report to CIC on services delivered, is the
centerpiece of the performance measurement component. It has been implemented
for two programs (RAP, LINC) and is in the pilot stage for the other two
programs (ISAP, Host), with full implementation expected for 2003/04.
The data being collected through iCAMS is generally quantitative.
A Performance Measurement Advisory Committee (PMAC), representing a range
of stakeholder groups and interests, has guided the process of implementing
performance measurement activities over the past two years.
CIC has also worked with stakeholders to address privacy and security
issues related to the collection of personal client information. CIC has
taken extensive measures to ensure that client information in iCAMS is
protected and being used in accordance with the provisions of the Privacy
Act and other relevant federal statutes and policies. More specifically,
CIC has implemented recommendations made by the Privacy Commissioner of
Canada; we have hired an independent third party to conduct a Privacy
Impact Assessment; and we have developed information and training tools,
such as a privacy pamphlet and poster informing clients of the purpose
of information collection in iCAMS, as well as a privacy manual for CIC
staff.
CIC has addressed security issues by hiring independent third parties
to conduct two Threat and Risk Assessments of iCAMS, developing security
requirements for service providers, and by ensuring that we use the up-to-date,
secure technologies.
Evaluation
This Evaluation Strategy will focus on the qualitative dimensions of
program assessment, while also making use of the quantitative data collected
through iCAMS. These qualitative dimensions include service quality, client
satisfaction, program design, and management factors. Evaluation frameworks
for each settlement programs were developed in 2000 by Goss Gilroy Inc.,
providing guidance on evaluation questions, indicators, methods, and data
sources. The documents also consider evaluation challenges, options, and
costing as well as performance reporting.
Contribution Agreement Process
The Contribution Agreement Process aims to develop and implement consistent,
comparable contribution relationships between CIC and service providers.
Settlement officers across the country are currently using standardized
templates for their agreements with service providers. These forms will
be updated on a regular basis.
Management Control Framework
The Management Control Framework will define and support good administrative
practices for CIC offices that deal with contribution programs. These
practices include staff training, monitoring, and auditing. CIC will tackle
this component in 2004/05.
Provincial-Territorial Accountability
This component of the Framework focuses on the accountability of provinces
that receive funding from the federal government, but manage their own
settlement programs. The goal is to ensure reasonably comparable services
across the country and comparable information that will contribute to
national program analysis.
DEVELOPING A CAF EVALUATION STRATEGY
CIC is ready to focus on the implementation of the CAF evaluation component.
A strategy for conducting settlement program evaluations will assist the
CAF Team in planning evaluation activities over the next two fiscal years.
CIC is proposing that the following steps be undertaken for each of the
settlement programs, starting with LINC in 2003/04 and then following
with ISAP and Host in 2004/05:
- Consult on settlement evaluation strategy
- Develop settlement evaluation communications strategy
- Gather information on settlement results statements and indicators
- Develop settlement results statements (program-level outcomes and
indicators)
- Review and validate settlement program evaluation frameworks
- Hire an external organization to conduct evaluation(s)
- Disseminate results of settlement evaluations and act on these results
Details on these steps are provided below and a timeframe is presented
in Attachment A.
1. Consult on settlement evaluation strategy
During February-April 2003, the CAF Team produced a draft evaluation
strategy and refined it with input from CIC colleagues. The strategy was
reviewed with the Performance Measurement Advisory Committee (PMAC) on
May 22-23, 2003. At the meeting, PMAC agreed to expand its mandate to
include evaluation, because of the strong links between evaluation and
performance measurement. PMAC members also made suggestions on the overall
strategy. These suggestions are reflected in this final document.
Timeframe: February 2003 - June 2003
2. Develop settlement evaluation communications Strategy
Regular communications with stakeholders is needed on the development
and implementation of the settlement evaluation strategy. A communications
strategy has been developed that makes frequent use of, among other mechanisms,
the CAF newsletter, the CAF website, and email communication. Please see
Attachment B for the Communications Strategy.
Timeframe: March - April 2003
3. Gather information on settlement results statements and indicators
CAT has gathered information from service providers, provincial partners,
other stakeholders, and countries on settlement results and indicators.
This survey will assist the CAF Team in putting together a draft set of
results statements and indicators for broad discussion. For example, we
are aware that the Governments of Alberta and British Columbia are currently
working with their provincial settlement partners to develop results statements
for their programs. We hope to benefit from the work currently being undertaken.
Please see Attachment C for a list of organizations
or documents that have been consulted on settlement results statements
and indicators. If you know of other organizations that should be consulted,
please contact a member of the Contribution Accountability Team.
Timeframe: May - June 2003
4. Develop settlement results statements and indicators
Based on the survey of results and indicators, the CAF Team will produce
a draft set of results statements and indicators, and will consult CIC
colleagues, PMAC, and a selection of all stakeholders on the draft statements.
Clients who have received services and are comfortable with being approached
will also be consulted on results statements and indicators. The CAF Team
will ask service providers to suggest appropriate clients.
The CAF Team will hire an independent, external RBM expert organization
to organize RBM workshops in each region with service providers and other
stakeholders. The purpose of these discussions will be to gather feedback
on the draft set of results statements and indicators. The CAF Team may
also attend already scheduled events such as the CCR meeting, TESL conference,
etc. to seek input on results statements. This will depend on the timeframe
for consultations. Additional feedback on results statements may be solicited
by telephone interviews or written surveys/questionnaires, if in-person
meetings or RBM workshops are not possible.
Input from service providers and stakeholders will be used by CIC to
finalize results statements and indicators for its settlement programs.
These statements and indicators will be clearly communicated to all interested
parties, and will be used as the basis for the formal evaluation.
Timeframe: July - October 2003 (LINC only)
5. Review and validate settlement program evaluation frameworks (2000)
The CAF Team will review, update, and summarize evaluation frameworks
prepared for CIC by Goss Gilroy Inc. in 2000. These frameworks contain
valuable information that was gathered through extensive internal and
external consultations. This information will need to be updated to reflect
changes in policies, processes, and lessons learned. Changes may also
be required in light of the results statements that will be articulated
through the collaborative process with stakeholders.
CIC will share the updated frameworks with PMAC and solicit their input
on specific issues, such as data collection methodologies and tools, evaluation
questions, scope, etc. Copies of the original evaluation frameworks may
be found on the CAF website: http://integration-net.ca/english/ini/caf-cipc/index.cfm.
We are recommending that the settlement program frameworks, developed
by Goss Gilroy, for each program be collapsed into one program evaluation
framework, with specific tools for different settlement programs included
as annexes where needed. This will facilitate a common approach across
programs and ensure programs contribute to overall CIC objectives and
expected outcomes, while at the same time respecting the differences among
the three programs.
Timeframe: September - November 2003 (LINC only)
6. Hire an external organization to conduct settlement evaluations
The evaluation framework, including stakeholder roles, evaluation tools,
and evaluation plan and schedule will be finalized and adopted. Terms
of Reference will be developed, and an organization will be hired competitively
to undertake the evaluations. Of the three settlement programs, as noted
above, LINC will be evaluated first.
Timeframe: October 2003 - March 2004 (LINC only)
7. Disseminate results of settlement evaluations and act on these results
Once the evaluations have been completed, CIC will disseminate results
of the evaluations to stakeholders and service providers. CIC will then
act on evaluation results. The information collected through the settlement
evaluations will form a solid baseline which will prove useful when undertaking
future evaluations.
Timeframe: March - May 2004 (LINC only)
RELATED ASSUMPTIONS, ISSUES, AND FACTORS
A number of assumptions and other factors need to be considered in finalizing
the evaluation strategy including:
Work Plan/Timelines
All settlement program evaluations should be completed by 2005. The LINC
evaluation will take place in 2003/04, with ISAP and Host scheduled for
2004/05.
Cost
Funds for implementing the CAF are limited. This is a particularly important
when considering a) the extent of the consultations on results statements
and indicators; and b) the scope of the evaluations.
Program Stakeholder Roles and Responsibilities
Effective program evaluation will depend upon a collaborative approach
and clear roles among settlement program stakeholders. It will be important
to clarify the roles that different stakeholders will play in evaluation
and how much time each stakeholder is able to devote to the process.
Currently, all key stakeholders are represented on PMAC. As proposed
earlier, PMAC's mandate should be expanded to cover both evaluation
and performance measurement. Input on evaluation activities should be
sought primarily through PMAC.
Current roles of various program stakeholders are described below with
some suggestions for modified responsibilities:
- CIC Contribution Accountability Framework (CAF) Team manages the development
and implementation of CAF and is composed of representatives from CIC
National Headquarters (NHQ) and regions. This group will develop the
settlement evaluation strategy, propose results for discussion by stakeholders,
and lead the review and validation of the settlement evaluation frameworks.
- CIC NHQ Program Managers provide guidance on the management of settlement
programs to ensure consistency and standard implementation across the
country. This group will provide input on the evaluation strategy and
its implementation.
- CIC Regional/Local CIC Personnel are involved in settlement program
management and implementation at different levels. They manage the contribution
agreements with service providers. All regions are represented on PMAC
by CIC representatives.
- CIC Priorities, Planning, and Research - Strategic Planning (PPR)
manages the development of Terms of Reference for conducting evaluations
and the formal evaluation process itself. PPR is also the departmental
liaison with the Treasury Board Secretariat, and will involve the CIC
Treasury Board Analyst, as needed.
- Service Providers act as collaborators in evaluation activities providing
information, data, and perspective based on their experience of the
programs. They should provide input to the development of results. Service
providers are represented on PMAC.
- Other Stakeholders, such as umbrella organizations, provincial partners,
and advocacy groups act as collaborators in evaluation, provide their
perspectives on the program from their involvement, and should provide
input to the development of results. Various stakeholders are represented
on PMAC.
- Program Participants/Clients are the beneficiaries of programs and
services and thus have a legitimate perspective for evaluation activities..
- Evaluators are the independent professionals who are retained through
a competitive process to conduct evaluations.
Attachment A
Attachment B
Attachment C
Resource Handbook
| 