National Initiatives

National Settlement Service and Standards Framework

Section 2
Program Standards

Program standards are minimum acceptable standards of quality that settlement services should attain to ensure a minimum acceptable level of service to clients, funders or professional bodies. These services should contribute to settlement outcomes for clients.

Although there are concerns among some agencies that standards may stifle creativity and flexibility, there seem to be several aspects that settlement programs across Canada may need to consider as their common minimum practice:

Regarding Clients
Program eligibility, admission procedure, client rights and responsibilities, confidentiality, and grievance, etc.

Regarding Service Delivery
Core activities of settlement service, time constraints, issues of cultural and linguistic appropriateness, case management, caseload, and follow-ups, etc.

Regarding Documentation of Service
What statistics are collected from clients with what common system; reporting system; storage of information, etc.

Regarding Program Outcomes
What level of outcomes (short-term, intermediate-term or long-term), what indicators, who should collect, and report to whom, etc.

This section deals mainly with the program-standards issues surrounding the measurement of outcomes and the methodology used for this measurement.

Using Results-Based Management

Results-Based Management is a participatory and team-based approach to management that seeks to focus on the efforts of an organization or program to achieve results. It involves defining realistic expectations, assessing risk, monitoring progress towards the achievement of results (performance measurement), integrating lessons learned into management decisions, and reporting them in simple, understandable ways. (CIC, 2003)

Program Logic Model

Governments and community-based settlement agencies have used a variety of Results-Based Management tools for planning, implementing, and monitoring, ranging from organic and experience-based tools to more formal tools. The program logic model is a formal tool used by a significant number of settlement agencies.

A program logic model is a systematic, visual way of describing a program and its planned results. In it the user describes and discusses program theory alongside program development. (Basic Guide to Outcome Based Evaluation in Non-Profit Organization)

The logic model, often referred to as the results chain, shows the casual or logical relationship between inputs, activities, outputs, and results. Although there are different possible configurations, the model can be visualized as follows:

Key Results Terminology

The terminology of Results-Based Management can be confusing, as similar terms are defined in slightly different ways. The definitions in the chart below conform to usage within the Treasury Board Secretariat of Canada.

Measuring Settlement Outcomes

Under ideal circumstances, an agency would have mechanisms for measuring its own performance and evaluating the outcomes of its services. After selecting appropriate indicators for measurement, the agency would adopt a strategy to track progress and would develop a process for regularly and systematically assessing the outcomes and cost-effectiveness of their programs and for making program adjustments and enhancements. Such assessments should be undertaken for programs individually, as well as in comparison with other programs in an agency.

Citizenship and Immigration Canada’s Contribution Accountability Framework (CAF) sees evaluation and performance measurement as two key accountability tools. CAF defines evaluation as the periodic, systematic collection and analysis of information on the performance of a policy, program or initiative to make judgements about relevance, progress, or success and cost-effectiveness and to inform future programming decisions about design and implementation. CAF defines performance measurement as the on-going collection of, and reporting on, information on program implementation to guide corporate decision-making.

The survey feedback indicates that most settlement agencies do not have the time and resource capacity required to conduct program evaluations. Nevertheless, the majority of agencies measure performance in a variety of ways and capture qualitative information about the impact of their programs.

OCASI/COSTI Partnership Project (June 1999) reports that the first human service organization in Ontario to adopt the use of standards was Family Service Ontario, an association of agencies providing counselling services through professional social workers. The association has used a voluntary accreditation system, which accredits agencies according to their compliance with standards for program delivery, governance and other areas.

The OCASI/COSTI Partnership Project (June 1999) further produced an extensive list of standards for service delivery, corresponding desired outcomes for clients and for an organization as a whole. Subsequently, using the framework, the Ottawa-Carleton Immigrant Services Organization developed standards for settlement services in all categories. The standards spell out specifically who does what, for whom, when, and why. A sample of the standards for services under the economic category is attached in Appendix A.

The importance of indicators

Before settlement agencies can measure the outcomes of their programs and services meaningfully, they must first define the indicators that measure the degree to which an immigrant is settled. However, it is not feasible to create a system that will measure settlement program outcomes with perfect precision.

The Auditor General of Canada’s 2000 Report on the Implementing Results-Based Management. Lessons from the Literature notes, “Over-complexity of a performance measurement system will lead to implementation problems and will simply frustrate stakeholders. The easier it is to use and apply, the more likely stakeholders will adopt and embrace the new approach.” It is therefore imperative for the settlement sector to measure a few indicators and keep data collection and administration simple.

When selecting performance indicators, choose those with the following qualities: (Shacter, 2002)

• Strong causal relationships with the ultimate outcomes;
• High level of program control over the indicators being measured;
• Show meaningful changes over time;
• Low cost in obtaining data; and
• Minimum amount of time required to obtain data or ease of administration.

At the current stage in the development of the settlement sector, it appears that service providers can be responsible for measuring mainly the immediate and intermediate outcomes of settlement services. It would be difficult for them to measure service outcomes that are beyond their control, and where causal relationships are uncertain. The survey made it clear that service providers have neither the financial and human resources nor the in-house expertise to conduct a longitudinal study to measure the medium-term or long-term impact of their services on immigrants.

Such a task would require dedicated resources so that information could be gathered from a range of services such as LINC, Host, and ISAP, and the outcomes would have to be studied over a considerable period of time, taking into account such variables as socio-economic trends, immigrant and refugee characteristics, and any changes in services and resources.

Current settlement evaluation initiatives

British Columbia has developed the Program Logic Model and Outcome Evaluation Framework for Settlement Information and Support Services (BC MCAWS, June 2003) on the next page. In addition, a provincial Advisory Committee on Performance Measurement and Reporting is developing a province-wide standardized questionnaire for immigrant-service agencies to use, in whole or in part, to evaluate their outcomes in British Columbia.

Since November 2002, the Community Programs Branch of Alberta Learning has been implementing the “Outcomes Measurement for Immigrant Settlement Services” Project. It provided training and coaching services to the Integrated Service Program (ISP) funders and funded agencies to understand the language and theory of Outcomes Measurement. It is now working toward the development of logic models and indicators for the ISP programs within each of the 17 immigrant serving agencies across Alberta.

CIC will be evaluating its settlement programs in 2003-04 and 2004-05. Adopting results-based management practices as a pre-requisite for meaningful program evaluations, CIC will therefore undertake an exercise to define program results and indicators in cooperation with stakeholders before proceeding with formal evaluations of its settlement programs.

Program Logic Model

INPUT
Resources dedicated to or consumed by the program

• Funding
• Staff
• Program and Office Space
• Materials and Equipment
• Community Partners
• Guest Speakers

ACTIVITIES
What the program does with inputs to fulfil its mission

OUTPUTS
The direct products of program activities

IMMEDIATE OUTCOMES
First benefits or changes in knowledge or ability resulting from program activities

MEDIUM-TERM OUTCOMES
Changes in attitude and behaviour that result from new knowledge or ability and link immediate outcomes to longer-term outcomes

LONG-TERM OUTCOMES
Ultimate outcomes a program desires to achieve or the most removed benefits that the program can reasonably expect to influence

Activity 1. Initial assessment and Action Planning

• Assess client needs & barriers
• Identify appropriate services
• Assist clients to set goals and priorities & develop realistic plans

Outputs:

• number of clients who received initial assessment and developed a plan of action toward settlement

Immediate Outcomes:

• Clients are aware of barriers and options

Medium-Term Ourcome(s):

• Medium-Term Ourcome #1

Activity 2. Orientation Information

Provide clients with an overview of Canadian services and systems, life and culture

Outputs:

• number of clients who obtained an overview of Canadian norms, culture, services, and systems

Immediate Outcomes:

• Clients understand Canadian norms and culture
• Clients are familiar with general services and systems

Medium-Term Ourcome(s):

• Medium-Term Ourcome #2

Activity 3. Enhanced Information and Adjustment Support

Provide clients with detailed information necessary to navigate specific processes, such as obtaining citizenship, family reunification, and making family and cultural adjustments

Outputs:

• number of clients who obtained assistance and detailed information to navigate a settlement, adjustment or immigration process

Immediate Outcomes:

• Clients know immigration, settlement and adjustment processes

Medium-Term Ourcome(s):

• Medium-Term Ourcome #2
• Medium-Term Ourcome #3

Activity 4. Referral

Introduce clients to relevant services and resources that meet their specific needs

Outputs:

• number of clients who obtained information and referral to a specific service or resource that meets their needs

Immediate Outcomes:

• Clients know about appropriate services and resources that meet their specific needs

Medium-Term Ourcome(s):

• Medium-Term Ourcome #3

Activity 5. Service Linking

Help clients use relevant services and resources and articulate their needs through

• appointment making
• form filling
• interpretation/ translation
• cross cultural orientation

Outputs:

• number of clients who were helped in finding a specific service

Immediate Outcomes:

• Clients can communicate their needs to appropriate services or resources

Medium-Term Ourcome(s):

• Medium-Term Ourcome #3
• Medium-Term Ourcome #4

Activity 6. Service Bridging

Help community service providers understand and serve immigrant clients

Outputs:

• number of service providers from the host community who obtained information or assistance from the settlement agency
• number and type of service partnerships with non-settlement service agencies

Immediate Outcomes:

• The general service community is more aware of immigrant needs

Medium-Term Ourcome(s):

• Medium-Term Ourcome #5

Activity 7. Service Support

Develop resources and expertise necessary to support service delivery

• networking and resource development
• participation in sectoral development
• professional development

Outputs:

• number and type of new community contacts and resource materials developed
• number and type of service-related community or government consultations attended
• number and type of professional development events or initiatives attended

Immediate Outcomes:

• The settlement agency has the necessary knowledge base and resources to deal with changing needs

Medium-Term Ourcome(s):

• Medium-Term Ourcome #6

Medium-Term Ourcomes

1. Clients develop realistic goals and plans
2. Clients adjust to Canadian life and culture and deal with issues that result from resettlement and adjustment
3. Clients meet personal and family needs and pursue improvements
4. Clients interact with mainstream society
5. The general service community adapts to newcomer needs
6. The settlement agency improves and enhances its services to meet changing needs

Long-Term Outcomes

1. Clients maintain a sense of well-being, confidence and independence
2. Clients contribute their skills and abilities to society
3. Clients identify with and enrich Canadian culture
4. Clients participate in and affect mainstream institutions and political processes

Outcomes Evaluation Framework

ACTIVITIES
See activities in logic model

IMMEDIATE OUTCOMES
First benefits or changes in knowledge or ability

MEDIUM-TERM OUTCOMES
Changes in attitude and behaviour that result from new knowledge or ability

INDICATORS
Observable and measurable characteristic or changes that signal that an outcome has been achieved

Activity 1. Initial assessment and Action Planning

Immediate Outcome

• Clients are aware of barriers and options

Indicators

• number and percentage of clients who report that they have learned of a situation (e.g. employment/ labour market situation) that might affect their settlement and know of options that are open to them
  
  Data Source: Clients
  Data Collection Method: Client survey

Activity 2. Orientation Information

Immediate Outcomes:

• Clients know Canadian norms and culture
• Clients know about general services and systems

Indicators

• number and percentage of clients who demonstrate knowledge of what is ordinary and socially acceptable in Canadian society
• number and percentage of clients who demonstrate knowledge of their legal rights and entitlements or responsibilities
• number and percentage of clients who demonstrate knowledge of major cultural events and holidays celebrated by the mainstream community
• number and percentage of clients who demonstrate knowledge of basic services or systems in the general community
  
  Data Source: Clients
  Data Collection Method: Client survey and test

Activity 3. Enhanced Information and Adjustment Support

Immediate Outcomes

• Clients know immigration, settlement or adjustment processes

Indicators

• number and percentage of clients who report that they now know the steps required to meet specific needs, such as status, family reunification, and benefits
  
  Data Source: Clients
  Data Collection Method: Client survey

Activity 4. Referral

Immediate Outcomes

• Clients know about appropriate services and resources that meet their specific needs

Indicators

• number and percentage of clients who report that they know how to contact a government or community service relevant to their need
  
  Data Source: Clients
  Data Collection Method: Client survey

Activity 5. Service Linking

Immediate Outcomes

• Clients can communicate their needs to appropriate services or resources

Indicators

• number and percentage of clients who report that the support they receive has improved their communication with a mainstream agency
  
  Data Source: Clients
  Data Collection Method: Client survey

Activity 6. Service Bridging

Immediate Outcomes

• The general service community is more aware of immigrants’ needs

Indicators

• number of mainstream organizations and businesses (contacted through service-bridging activity) that report a better understanding of immigrant needs
  
  Data Source: Mainstream agencies
  Data Collection Method: Interview with mainstream agencies

Activity 7. Service Support

Immediate Outcomes

• The settlement agency has the necessary knowledge base and resources to deal with changing needs

Indicators

• number and percentage of settlement practitioners who report that they have sufficient resources, training and support to do their job
  
  Data Source: Settlement staff
  Data Collection Method: Staff survey

Medium-Term Ourcome #1: Clients develop realistic goals and plans

Indicators

• number and percentage of clients who report that they have identified or redefined their employment plans or personal goals based on an increased awareness of barriers or options
  
  Data Source: Clients
  Data Collection Method: Client survey

Medium-Term Ourcome #2: Clients adjust to Canadian life and culture and deal with issues that result from resettlement and adjustment

Indicators

• number and percentage of clients who report a change in lifestyle, attitude or practice that reflect Canadian norms culture such as parenting, gender roles, social recreational activities, banking and budgeting
  
  Data Source: Clients
  Data Collection Method: Client survey or focus group
  
• number and percentage of clients who report that they have used appropriate services to deal with legal, interpersonal or family issues that result from resettlement or adjustment
  
  Data Source: Clients, Settlement staff
  Data Collection Method: Client survey, Service Record

Medium-Term Ourcome #3: Clients meet personal and family needs and pursue improvements

Indicators

• number and percentage of clients who report that they have used mainstream services and systems to meet settlement needs such as shelter, status, documentation, employment and educational counselling, day care, or to obtain basic entitlements such as social benefits and health care
  
  Data Source: Clients, Settlement staff
  Data Collection Method: Client survey, Service Record
• number and percentage of clients who report that they have used a service or program to improve their situation or quality of life or to attain personal goals such as recognition of credentials, or enrollment in educational or training programs
  
  Data Source: Clients, Settlement staff
  Data Collection Method: Client survey, Service Record

Medium-Term Ourcome #4: Clients interact with mainstream society

Indicators

• number and percentage of clients who report that they participate or volunteer in community or school activities or networks
  
  Data Source: Clients, Settlement staff
  Data Collection Method: Client survey, Service Record

Medium-Term Ourcome #5: The general service community adapts to newcomer needs

Indicators

• number and percentage of mainstream organizations and businesses (contacted through service bridging activity) that adapt a product, service process or programming, policy or practice to take account of immigrant needs
  
  Data Source: Settlement agency or Mainstream agencies
  Data Collection Method: Settlement agency service record or Interview with mainstream agencies

Medium-Term Ourcome #6: The settlement agency improves and enhances its services to meet changing needs

Indicators

• Agency reports that they have aligned their service activities and approach with changing needs and environment
  
  Data Source: Settlement agency
  Data Collection Method: Agency report

Index Next...