The Government of Canada's Policy on Evaluation[ 1 ] took effect on April 1, 2009 and replaces the 2001 Policy on Evaluation. This renewed policy has the objective of creating a comprehensive and reliable base of evaluation evidence to support policy and program improvement, expenditure management, Cabinet decision-making and public reporting. It introduces a number of changes intended to improve the neutrality, timeliness and quality of evaluations. These include a focus on core issues of value for money (i.e., program relevance and performance) and expanded coverage to include evaluation of all program spending every five years, effective March 31, 2013.
The Policy on Evaluation requires that credible, timely and neutral information on the ongoing relevance and performance of direct program spending is:
The policy directly supports the Expenditure Management System of the Government of Canada by ensuring comprehensive and systematic information on program relevance and performance is available to support decision-making and the strategic review process. It also supports a requirement of the Financial Administration Act (as amended by the Federal Accountability Act) to evaluate all ongoing grants and contribution programs every five years.
This charter explains how the Canada Border Services Agency (CBSA) will implement the Policy on Evaluation. It summarizes the mandate, governance structure, operating standards and significant relationships of CBSA's evaluation function. It outlines the adherence of the evaluation function to professional standards for evaluation, as set out in the policy and by other recognized professional bodies, such as the Canadian Evaluation Society.
This charter has been prepared to inform all CBSA employees about the evaluation function, and the support and co-operation expected from them as evaluation work is undertaken.
The CBSA is a department of the federal government within the meaning of Schedule 2 of the Financial Administration Act. As such, it is subject to the application of the Treasury Board Policy on Evaluation issued pursuant to paragraph 7(1)(a) of the Financial Administration Act[ 2 ]. The policy requires that deputy heads establish a robust, neutral evaluation function in their department and ensure that their department adheres to this policy and to its supporting directive and standard.
The Treasury Board's Directive on the Evaluation Function supports the objectives of the Policy on Evaluation by operationalizing requirements for departmental staff involved in evaluation and establishing the specific requirements to enable the production of credible, timely, neutral, and cost-effective evaluations to support policy, program and expenditure decision-making[ 3 ].
Our Mission is as follows:
To support the achievement of CBSA's strategic priorities by providing senior management with independent, objective and evidence-based information, advice and guidance on program relevance, performance and efficiency for effective decision-making.
Our Vision is:
To be a capable and reliable partner to CBSA management and be widely recognized for the quality and impact of our evaluation work.
Our work is guided by the following CBSA values:
CBSA's evaluation function is delivered by the Program Evaluation Division, as part of the Internal Audit and Program Evaluation Directorate. It reports to the Agency's Executive Evaluation Committee (EEC), which is chaired by the President.
The evaluation function's role is to inform management decisions with objective, evidence-based analysis on the efficiency, effectiveness and relevance of CBSA policies, programs and initiatives.
The Policy on Evaluation requires that the Agency develop a rolling five-year evaluation plan that demonstrates evaluation coverage of all direct program spending over a five-year period. The evaluation universe, based on the Agency's Program Activity Architecture (PAA), provides the cornerstone for evaluation planning within the CBSA.
The CBSA Five-Year Evaluation Plan is presented to the EEC for approval. It provides:
Resource requirements identified in the plan form the basis for human resource planning for the Division.
The evaluation universe presents the full set of evaluable programs[ 4 ] within the CBSA based on the Agency PAA. However, the plan does not represent a one-to-one mapping of evaluations to PAA activities or sub-activities. Program activities may be divided or grouped together, as required, to ensure that evaluations are of a manageable size, while at the same time providing adequate coverage to meet the requirements of the Policy on Evaluation.
While the evaluation plan must demonstrate coverage of all program spending on a five-year cycle, a number of factors are taken into consideration in determining the timing of each evaluation. They include:
The plan must also reflect the need for flexibility so that the Program Evaluation Division can respond to requests for one-off evaluation studies, and for advice and guidance on results-based management, including the development of performance measurement strategies and special studies.
CBSA Vice-Presidents, direct reports to the President, and Directors General are consulted to validate the proposed evaluation groupings and project timing. These consultations also provide an opportunity to get their perspectives on priority areas and specific issues that need to be addressed by each study.
The draft CBSA Five-Year Evaluation Plan is presented for review and approval by the EEC on an annual basis.
The evaluation function, through the RBM Unit, provides results-based management expertise and guidance to Agency management and works to foster a results-based management culture. The unit also provides advice and guidance to support research activities undertaken by other parts of the Agency, including post-implementation reviews.
RBM is a life-cycle approach to management that integrates strategy, resources, processes, and measurements to improve decision-making, transparency and accountability. The approach focuses on achieving outcomes, implementing performance measurement systems and reporting on performance. The RBM Unit's activities focus on the following areas:
The unit supports the division's evaluation projects by developing key program evaluation components – the program logic model and related performance measurement framework (PMF)[ 5 ], in advance of conducting an evaluation.
PMSs[ 7 ] are intended to help program managers continuously monitor and assess program results, economy and efficiency. The RBM Unit provides advice and guidance to program management as they develop and implement these strategies. For Programs Branch, this includes collaboration and consultation with its Performance Monitoring Division. Once developed, these strategies are approved by the Directors General responsible for the delivery of the programs, the Agency's Senior Financial Officer, and the Head of Evaluation.
The RBM Unit provides a single window into the CBSA for the coordination of activities associated with evaluations of horizontal initiatives led by other departments and agencies. The vast majority of these evaluations are conducted by consultants, with direction and input provided through advisory or steering committees comprised of representatives from the partner agencies and departments. To ensure that Agency activities are accurately reflected in the evaluation study, CBSA program management is represented on these committees and facilitates the provision of pertinent data and information to the evaluation team. Evaluation officers represent the Program Evaluation Division on these committees, facilitate and participate in CBSA interviews, and review data and draft versions of evaluation reports. The RBM Unit coordinates Agency approval of horizontal evaluation reports, including the CBSA's management response to recommendations.
Independence and objectivity in reporting on program relevance and performance are keys to the success of the evaluation function in achieving its objectives and adding value. This requires support and co-operation from several key players. The roles of the principal players are summarized below.
In accordance with Treasury Board policy, the President of the CBSA has the responsibility for establishing a robust and neutral evaluation function and for establishing a departmental evaluation committee comprised of senior officials.
Chaired by the CBSA President, the EEC membership includes the Executive Vice-President and all Agency Vice-Presidents, Associate Vice-Presidents and direct reports to the President. The evaluation function reports on its activities to the committee, which plays an important oversight role.
Principal responsibilities of the EEC include:
The Program Evaluation Division serves as the Secretariat for the EEC. The following protocol has been established for the committee:
A relationship of mutual trust and respect between employees who carry out the evaluation function and other CBSA staff is critical to achieving the objectives of the evaluation function and ensuring effective program management support.
While evaluation projects are being conducted, CBSA management is expected to be co-operative and reasonably available to engage in evaluation activities. Agency management must respond in a timely manner to evaluation staff members' information requests and ensure access to the reliable, timely and objective information needed to conduct their evaluations. All evaluation employees are required to have secret security clearance and, while respecting clearance requirements, they will have unlimited access to all Agency documents.
In addition to these general responsibilities, Agency management is expected to:
The Director General of the Internal Audit and Program Evaluation Directorate is the Agency's Head of Evaluation and is responsible for ensuring an effective evaluation function.
Reporting to the Director General of the Internal Audit and Program Evaluation Directorate, the Director, Program Evaluation Division, is the senior full-time Evaluation Executive in the Agency. This individual has been delegated responsibility by the Head of Evaluation for ensuring that all evaluation activities adhere to applicable policies and standards. The Director is also responsible for the day-to-day operation of the Program Evaluation Division and for verifying overall quality assurance of all evaluation products before they are presented to the Agency's EEC for approval.
The following section provides a high-level description of the evaluation function's operational process.
The following process map provides a graphical representation of the operational process.
Employees carrying out the evaluation function have an obligation to perform the work objectively, professionally, and in a spirit of mutual trust and respect. The following sections set out the standards for the evaluation function that all CBSA staff can expect. These standards embrace the well-defined guiding principles set out in the Policy on Evaluation, and by Treasury Board and professional evaluation bodies, in particular, the Canadian Evaluation Society and the American Evaluation Association.
Evaluation projects will be free from interference from program management in the determination of the scope of work, in the conduct of the work and in the communication of the results.
Evaluators will be objective in performing their work, have an impartial and unbiased attitude, and avoid conflicts of interest. Evaluators will act with integrity in their relationships with all stakeholders and behave in a manner that is consistent with the Values and Ethics Code for the Public Service and the CBSA's values of integrity, respect and professionalism.
Evaluators will possess the knowledge, skills and other competencies needed to perform their responsibilities. Evaluators will apply due diligence in the performance of their work. Evaluation professionals collectively involves either using the required knowledge, skills and other competencies to complete the work or taking steps to obtain these elements, as required.
Evaluators will comply with government policies related to information collection, use, preservation and dissemination. Stakeholders and participants in an evaluation will be informed of the level of confidentiality and privacy that is afforded them under the Privacy Act[ 8 ].
Evaluators will perform their responsibilities professionally and courteously, remaining sensitive to the additional workload imposed on clients by the evaluation.
Evaluation reports will respect all relevant Treasury Board standards, and will be written in a clear and concise manner. Reports will present conclusions and recommendations that flow logically from evaluation findings and will contain clear and actionable recommendations and timing for the action plan.
Completed reports, including management responses and action plans, will be presented to the EEC for discussion and approval. Prior to seeking EEC approval, draft evaluation reports will be provided for comment to senior management responsible for the area under evaluation. Notwithstanding this step in the process, should a serious issue be discovered over the course of the evaluation research, the evaluators would raise it immediately with senior management.
Reports will be considered final after they have been approved by the EEC, and will be released to the public after proofreading and final copy editing by the CBSA Communications Directorate. Reports in both official languages are posted on the CBSA Web site, with a link to the reports posted on the Treasury Board Web site. CBSA ATIP officials are consulted before the reports are released to ensure they respect guidelines on disclosure of information.
Timely evaluation reports are an important source of information for senior CBSA officials on significant issues of effectiveness, efficiency and management practices. The following time frames have been established for the completion of evaluation reports:
Unless a PMS is required by Treasury Board, there are no timelines associated with their completion as these are the responsibility of program management. However, the Program Evaluation Division will respond to formal requests from Directors General for such guidance and assistance within five working days to determine how to move the requests forward.
This charter is issued under the authority of the President of the CBSA and approved by the EEC. The charter came into effect on April 1, 2011.
The Head of Evaluation will recommend updates to the charter as required. Updates will be effective upon approval of the EEC.