Species at Risk Program: Results-Based Management and Accountability Framework and Risk-Based Audit Framework

5. Performance and Risk Monitoring and Reporting

Performance and risk will be monitored as an integrated strategy that relies on the results logic shown in Section 3 and the risk assessment shown in Section 4.

5.1 Performance Indicators and Measurement Strategy

Corresponding to the SAR Program outputs and expected outcomes (see Section 3) is a set of performance indicators and a measurement strategy, which have been presented in Table 13. A number of indicators are relevant for both performance measurement of results and risk. These "dual" indicators are shown in bold and are a key efficiency feature of an integrated RMAF-RBAF. Note that risk indicators are linked to specific risks (as identified and numbered in Section 4).  

For each output and outcome, the measurement strategy identifies

  • indicators;
  • targets;
  • those accountable for delivering on the outputs and the outcomes;
  • data collection methods and / or sources of information;
  • those responsible for collecting, analyzing and reporting; and
  • frequency of data collection.

Indicators

Quantitative and qualitative indicators have been identified strategically in order to demonstrate progress in producing the outputs and achieving the outcomes. The ability to collect data was an important factor in choosing the indicators.

Targets

Some indicators have associated targets. These targets are presented in Table 13 under the Targets column. Other performance targets will be identified through an implementation review (2009) (see Section 8 RMAF-RBAF Implementation Review Strategy) and after gathering benchmarking data (2008-2009). However, it is important to note that a number of indicators do not lend themselves to performances targets.

Accountability

Those accountable from the core departments for producing the outputs and achieving the set outcomes have been identified at the Assistant Deputy Minister and Director General levels, depending on the department/agency. Partners, Aboriginal people and stakeholders have also been identified as accountable for certain outcomes, given that their involvement in the Program is critical to the success of delivering on the species at risk results.

Data Source and Evaluation Methods

The data sources for most of the output indicators and for several of the outcome indicators are Program management databases and Program files. This information will be used to support the management of the Program as well as future evaluations. The sources for the remaining indicators are various evaluation methodologies (e.g., document review, interviews, surveys, case studies and/or workshops).

Responsibility for Data Collecting, Analyzing and Reporting

Those responsible from the core departments for the data collection, analysis and reporting have been identified at the Director level.

Frequency of Data Collection

The majority of the outputs indicators and several outcome indicators will be tracked and reported annually. The remaining indicators will require a greater investment in data collection and assessment methodologies. Most of these indicators will be the responsibility of the evaluation team (supported by the core departments).

It is important to note that formal risk evaluations or assessment will be done on a periodic basis; however, some indicators of risk may be monitored on an ongoing basis as part of good management practices. This ongoing monitoring is critical to Program success as factors that affect risk levels / exposure can change quickly and, as a result, new or modified mitigation strategies may be needed. 

Table 13: Menu of Core Indicators (Performance and Risk) and Assessment Measurement Strategy for SAR Program12 13
Performance AreaKey Indicators
(Note: Bolded entries identify indicators of both risk and performance and bold italics identify indicators related to core departments’ workloads)
TargetsAccoun-
tability for Results
Data Source /  Eval’n MethodsRespon-sibility for Collecting DataReporting
Assessment / Protection
Key Outputs
COSEWIC Status Reports
  • % of SAR species for which COSEWIC completed a conservation status review within 10 years.
100%COSEWICSpecies at Risk (SAR) Database14EC Director Conservation Service Delivery and Permitting (CSDP)Annual
Secretariat support to COSEWIC
  • Trend in the resources provided to COSEWIC activities
    • # of meetings organized
 EC Associate ADM ESBSecretariat filesEC Director CSDPAnnual
Report on the general status of species
  • Trend in the number of species addressed in report on the general status of species
 DFO ADM O&SG and Regional DGs

EC Associate ADM  ESB

PC DG NP
Wildlife Species Website15EC Director CSDP, PC Executive Director EI Annual
Federal SAR assessment policies and guidelines
  • Extent to which core departments follow the policies and guidelines (6)
  • Extent to which policies and guidance documents meet Program management needs (6)
 DFO ADM O&SG and Regional DGs

EC Associate ADM ESB

PC DG NP
Outcome Evaluation (document review, interviews)Evaluators2010- 2011
Listing consultations
  • % of species assessed for which consultation have taken place (1, 2, 3, 4)
  • % of species assessed found on Aboriginal lands for which consultation have taken place and have met the federal requirements, including those of land claim agreements (2, 4)
1) 100%
2) 100%
DFO ADM O&HS and Regional DGs

EC Associate ADM  ESB
SAR DatabaseDFO Director Species at Risk Secretariat (SARS), EC Director CSDP,Annual
  • Number of partners contacted during listing consultations (1, 4)
  • Number of Aboriginal people or organizations contacted during listing consultations (2, 4)
  • Number of stakeholders contacted during listing consultations (3, 4)
 DFO ADM O&HS and Regional DGs

EC Associate ADM  ESB
Departmental RecordsDFO Director SARS,  EC Director CSDP Annual
Permits

Statements of rationale
  • Number of permit requests or applications received (4)
  • Number of permits issued (4)
  • Average time required to respond to permit applications, trend in average (4)
  • % of enforcement actions with respect to permits
 DFO ADM O&HS,  Regional DGs and SO, SARA

EC Associate ADM  ESB and CEO

PC DGs E, and W&N
DFO Permitting Database

EC SAR Permitting Database

PC Permitting Database
DFO Director SARS, EC Director CSDP and PC Executive Director EIAnnual
Compliance promotion strategies, plans, tools and policies
  • % of annually listed SAR covered by compliance promotion strategies and plans16 (4)
  • % of regulated communities that have been targeted by compliance promotion activities
1) 100%DFO ADM O&HS and Regional DGs

EC Associate ADM  ESB

PC DG NP
Compliance strategies and
compliance promotion plans

SAR Database
DFO Director SARS, EC Director CSDP and PC Executive Director EIAnnual
Enforcement tools and policies
  • % of listed SAR covered by compliance promotion and enforcement planning mechanisms (4)
 DFO ADM O&HS, Regional DGs and SO, SARA

EC Associate ADM  ESB, CEO

PC DG NP
National Inspection Plans

SAR Database
DFO Director SARS, EC Director CSDP and PC Executive Director EIAnnual
Trained enforcement officers
  • Number of SARA-trained enforcement officers (F/P/T) (4)
 DFO SO, SARA

EC CEO

PC DG NP
DFO Enforcement Database

EC NEMESIS

PC Enforcement Database
DFO SO, SARA

EC Director Wildlife Enforcement

PC DG NP
Annual
Enforcement activities / investigative reports
  • Number of recorded violations, charges laid, charges approved, enforcement inspections and investigations by the core departments' enforcement officers (4)
  • Number of S.93 requests for investigation processed by the core departments' enforcement officers (4)
  • Number of S.93 requests for investigation completed by the core departments' enforcement officers (4)
 DFO SO, SARA

EC CEO

PC DGs E and W&N
DFO Enforcement Database

EC NEMESIS

PC Enforcement Database
DFO SO, SARA

EC Director Wildlife Enforcement

PC DG NP
Annual
Federal SAR protection policies and guidelines
  • Extent to which core departments follow the policies and guidelines (6)
  • Extent to which policies and guidance documents met program management needs (6)
 DFO ADM O&HS, Regional DGs

EC Associate ADM  ESB

PC DG NP
Outcome evaluation (interviews, document review)Evaluators2010- 2011
Response statements
  • % of response statements issued within 90 days (6)
100%DFO ADM O&HS

EC Associate ADM  ESB
SAR DatabaseEC Director CSDPAnnual
Minister’s recommendations

Statements that support decisions
  • % of GIC receipt that led to a GIC decision issued within nine months (6)
 DFO ADMO&HS EC Associate ADM  ESBSAR DatabaseEC Director CSDPAnnual
GIC listing order
Orders to legally protect critical habitat
  • Number of orders to protect critical habitats(4)
  • % of critical habitat in federal protected areas that is published in the Canada Gazette on time as per subsection 58(3)
2) 100%DFO ADM O&HS

EC Associate ADM ESB

PC DGs E and W&N
SAR Database / Recovery Information Management System (RIMS)EC Director CSDPAnnual
Ministerial opinions on effective protection
Orders, other than listing orders
  • Number of emergency listing orders, trend in number of orders (4)
  • Number of emergency protection orders (4)
  • Number of orders to apply section 32 and/or section 33 to non-federal lands (safety net) (1, 4)
 DFO ADM O&HS

EC Associate ADM ESB
SAR DatabaseEC Director CSDPAnnual
Regulations to protect critical habitat on federal lands
  • Number of regulations to protect critical habitat on federal lands (4)
 DFO ADM O&HS

EC Associate ADM ESB
SAR DatabaseEC Director CSDPAnnual
Immediate Outcomes
Engagement by partners and Aboriginal people in SAR assessment and protection activities
  • Extent to which partners are engaged in assessment and protection activities (1)
  • Extent to which Aboriginal people are engaged in assessment and protection activities (2)
 DFO ADM O&HS, Regional DGs

EC Associate ADM  ESB

PC DG E and W&N

Partners and Aboriginal people
Outcome Evaluation (survey)Evaluators2010- 2011
  • Number of partners providing input during the listing process (1)
  • Number of Aboriginal people or organization providing input during the listing process (2)
 Partners and Aboriginal peopleDepartmental RecordsDFO Director SARS, EC Director CSDPAnnual
Identification of SAR priorities through a coordinated early detection system based in science, ATK and risk ranking of species
  • % of species assessed by COSEWIC that were identified a priori as at risk in the report on the general status of species
  • % of COSEWIC decisions that were informed by ATK  and scientific information
 COSEWICSAR DatabaseEC Director CSDPAnnual
Legislative frameworks collectively provide protection to species at risk and their residence and critical habitat
  • Number of times SARA was invoked to protect species at risk and their residence on lands of federal responsibility (1)
  • Number of times SARA was invoked to protect critical habitat identified in recovery strategy (1)
 DFO ADM O&HS

EC Associate ADM  ESB

PC DG E and W&N

Partners and Aboriginal people
SAR DatabaseEC Director CSDP, PC Executive Director EIAnnual
  • Progress towards a complementary set of F/P/T species at risk legislation and policies (1, 6)
 DFO ADM O&HS

EC Associate ADM  ESB

PC DG NP

Partners and Aboriginal people
Outcome evaluation (document review, interviews)Evaluators2010- 2011
Legal obligations are met
  • Number of legal challenges to the listing process (6)
  • % of legal challenges successfully defended or resolved (6)
 DFO ADM O&HS

EC Associate ADM  ESB
EC Director General’s office filesEC Director CSDPAnnual
Implementation of compliance promotion
  • Implementation of actions outlined in compliance promotion plans
 DFO ADM O&HS and Regional DGs

EC Associate ADM ESB

PC DG ER&VE
Compliance promotion plansEvaluators2010- 2011
Enforcement of general and critical habitat prohibitions by departments
  • Number of recorded violations, charges laid, charges approved, enforcement inspections and investigations by core departments enforcement officers (4)

  • % of S.93 of SARA requests for investigation processed by core departments enforcement officers (4)

  • % of S.93 of SARA requests for investigation completed by core departments enforcement officers (4)
 DFO SO, SARA

EC CEO

PC DG NP
DFO Enforcement Database

EC NEMESIS

PC Enforcement Database
DFO SO, SARA

EC Director Wildlife Enforcement

PC DG NP
Annual
Intermediate Outcomes
Engagement and support by stakeholders in the development of SAR assessment and protection activities
  • Extent to which stakeholders are engaged in assessment and protection activities (3)
 DFO ADM O&HS and Regional DGs

EC Associate ADM ESB

PC DGs E and W&N

Stake-holders
Outcome Evaluation (survey)Evaluators2010- 2011
  • Number of stakeholders providing input during the listing process (3)
 Stake-holdersDepartmental RecordsDFO Director SARS, EC Director CSDP Annual
Critical habitat is protected
  • % of critical habitat identified in SARA recovery strategies that is protected
 DFO ADM O&HS and Regional DGs

EC Associate ADM ESB

PC DGs E and W&N

Partners and Aboriginal people
SAR DatabaseEC Director CSDP, PC Executive Director EIAnnual
Recovery Planning
Outputs
Recovery Strategies

Recovery Action Plans

Management Plans
  • % of recovery strategies, action plans and management plans developed and published within legislative timelines for listed SAR (4,5)
  • % of listed SAR with a recovery strategy, action plan or management plan in place (4)
  • % of threatened and endangered species for which critical habitat was identified in recovery strategies or action plans  (part or whole)
1) 100%DFO ADM O&HS and Regional DGs

EC Associate ADM  ESB

PC DGs E and W&N
DFO Recovery Records

SAR Database  / RIMS

PC Recovery Records
DFO Director SARS

EC Director CSDP

PC Executive Director EI
Annual
Recovery Planning Consultations
  • % of listed SAR for which recovery planning consultations have taken place (1, 2, 3, 4, 5)
  • Number of partners and Aboriginal contacted during the recovery planning consultations (1, 2)
  • Number of stakeholders contacted during the recovery planning consultations (3)
1) 100%DFO ADM O&HS and Regional DGs

EC Associate ADM ESB

PC DGs E and W&N
SAR Database / RIMSEC Director CSDP, PC Executive Director EIAnnual
Federal SAR recovery planning policies and guidelines
  • Extent to which core departments follow the policies and guidelines (6)
  • Extent to which policies and guidance documents meet program management needs (6)
 DFO ADM O&HS and Regional DGs

EC Associate ADM ESB

PC DG NP
Outcome Evaluation (document review, interviews)Evaluators2010- 2011
Immediate Outcomes
Engagement by partners and Aboriginal people in SAR recovery strategies, action plans and management plans
  • Extent to which partners are engaged in recovery planning activities (1) a
  • Extent to which Aboriginal people are engaged in recovery planning activities (2)
 DFO ADM O&HS and Regional DGs

EC Associate ADM ESB

PC DGs E and W&N

Partners and Aboriginal people
Outcome Evaluation (survey)Evaluators2010- 2011
  • Number of partners that have participated during recovery planning (1)
  • Number of Aboriginal people or organizations that have participated during recovery planning (2)
  • Number of provincial recovery documents that can be adopted in their present condition (1)
 Partners and Aboriginal peopleDepartmental Records

SAR Database / RIMS

DFO Director SARS

EC Director CSDP

PC Executive Director EI

Annul
Intermediate Outcomes
Engagement and support by stakeholders in recovery strategies, action plans and management plans 
  • Extent to which stakeholders are engaged and support recovery planning (3)
 DFO ADM O&HS and Regional DGs

EC Associate ADM ESB

PC DGs E and W&N

Stake-holders
Outcome Evaluation (survey)Evaluators2010- 2011
  • Number of stakeholders that provided input during recovery planning (3)
  • % of recovery planning documents for which comments were received through the public comment period from stakeholders (3)
 Stake-holdersDepartmental RecordsDFO Director SARS

EC Director CSDP

PC Executive Director EI
Annual
Implementation
Outputs
Core departments’ implementation of priority actions
  • % of priority actions implemented and completed (4, 5)
 DFO ADM O&HS and Regional DGs

EC Associate ADM ESB

PC DGs E and W&N
5-year  recovery implementation reviewDFO Director SARS

EC Director CSDP

PC Executive Director EI
Beginning 2011
G&C Funding Support

Funding agreements
  • Number of applications for funding received (4)
  • Trend in the amount of G&C funding provided
  • Number of projects funded under G&Cs
  • Number of funding agreements signed (4)
  • Number of projects funded under the Aboriginal programs (2)
  • Number and type of Aboriginal organizations involved in the Aboriginal programs (2)
 DFO ADM O&HS

EC Associate ADM ESB

PC DG E and W&N

Partners, Aboriginal people and stakeholders
DFO Funding Database

EC SARA Funding Program Database

RENEW Annual Reports
DFO Director SARS

EC Director CSDP
Annual
CEAA recommendations
  • Number of CEAA projects that identified mitigation and monitoring measures related to SAR
  • Number of projects reviewed by an environmental assessment officer that had a SAR component
 DFO ADM O&HS and Regional DGs

EC Associate ADM ESB

PC DGs E and W&N
CEAA RegistryEC Director CSDP, PC Executive Director EIAnnual
Federal SAR implementation policies and guidelines
  • Extent to which core departments follow the policies and guidelines (6)
  • Extent to which policies and guidance documents meet program management needs (6)
 DFO ADM O&HS and Regional DGs

EC Associate ADM ESB

PC DG NP
Outcome evaluation (interviews, document review)Evaluators2010- 2011
Immediate Outcomes
Implementation of priority  recovery actions by partners and Aboriginal people
  • Distribution of the financial contribution by types of projects  (IRF, HSP, Aboriginal, ESRF) funded by the federal government in recovery implementation
 DFO ADM O&HS

EC Associate ADM  ESB

PC DGs E and W&N
Departmental Financial RecordsDFO Director SARS

EC Director CSDP
Annual
  • % of priority actions identified in recovery strategies, action plans and management plans that have been implemented by partners and Aboriginal people(1, 2)
  • % of projects funded under G&Cs that have been implemented (1, 2)
 Partners and Aboriginal peopleSAR Database / RIMS / SARA Funding Program Database
Increased Aboriginal capacity to participate in SAR planning and implementation
  • Level of awareness of Aboriginal people with respect to SAR issues and opportunity for action (2)
 Aboriginal people

EC Associate ADM  ESB
Outcome Evaluation (findings analyses)Evaluators2010- 2011
Inclusion of SAR, their residence and /or critical habitat in environmental assessments
  • Number of CEAA projects that identified mitigation and monitoring measures related to SAR
 DFO ADM O&HS and Regional DGs

EC Associate ADM ESB

PC DGs E and W&N
CEAA RegistryDFO Director SARS

EC Director CSDP

PC Executive Director EI
Annual
Intermediate Outcomes
Stakeholders have the information necessary to contribute to the protection of species at risk and their habitat
  • Level of awareness of stakeholders with respect to SAR issues and opportunity for action (3, 8)
  • Effectiveness of compliance promotion tools and policies in encouraging related communities to comply with SARA requirements
 DFO ADM O&HS and Regional DGs

EC Associate ADM ESB

PC DG ER&VE

Stake-holders
Outcome evaluation (project files, interviews, surveysEvaluators2010- 2011
Implementation of priority recovery actions by stakeholders
  • % of actions identified as priority in recovery strategies, action plans and management plans that have been implemented by stakeholders (3)
 Stake-holdersDFO Recovery Records

SAR Database / RIMS

PC Recovery Records
DFO Director SARS

EC Director CSDP
Annual
Monitoring and Evaluation
Outputs
SAR 5-year  recovery implementation reports
  • % of reports posted on Public Registry on time (4)

 

100%DFO ADM O&HS

EC Associate ADM  ESB

PC DGs E and W&N
SARA RegistryEC Director CSDPBeginning 2011
Annual report to Parliament on the administration of SARA
  • Extent to which requirements under the Act are met (5)
 DFO ADM O&HS and Regional DGs

EC Associate ADM  ESB

PC DG NP
Outcome Evaluation (findings analyses)Evaluators2010- 2011
Federal SAR monitoring and evaluation policies and guidelines
  • Extent to which core departments follow the policies and guidelines (6)

 

 DFO ADM O&HS and Regional DGs

EC Associate ADM ESB

PC DG NP
Outcome Evaluation (document review, interviews)Evaluators2010- 2011
Round table meetings

Responses to roundtable recommendations
  • Timeliness of roundtable meetings
  • Timeliness of Minister’s response to recommendations
1) 2 yearsDFO ADM O&HS

EC Associate ADM  ESB

PC DG NP
SARA RegistryEC Director CSDP Bi-annual
Immediate Outcomes
Improved species monitoring
  • % of Schedule 1 species that are monitored
  • % of species with monitoring programs that cover critical habitat (7)
  • Reduction in the number of data deficient species (9)
  • # of collaborative arrangements for sharing data on SAR (7)
 DFO ADM O&HS and Regional DGs

EC Associate ADM ESB

PC DGs E and W&N

Partners, Aboriginal people and Stake-holders
SAR Database

 Other  databases
DFO Director SARS

EC Director CSDP

PC Executive Director EI
Annual
Improved administration of SARA
  • Extent to which round table recommendations have an influence over the management of the Program
  • Examples of improvements to the administration of SARA (4), such as:
    • functioning of governance structures
    • % of deadlines that are met (5)
 DFO ADM O&HS

EC Associate ADM ESB

PC DG NP
Outcome Evaluation (document review, interviews)Evaluators2010- 2011
Intermediate Outcomes
Engagement and support for species monitoring
  • Level of engagement and support by stakeholders in species monitoring (3, 7)
 Stake-holdersOutcome Evaluation (interview, document review)

SAR 5-year Recovery Implementation Reports
DFO Director SARS

EC Director CSDP
Annual
SAR Program Long-Term Outcomes
Conservation and protection of species at risk
  • Number of species assessed by COSEWIC that are subsequently listed under SARA
  • Number of species that have been reassessed to a lower threat category
  • Number of species that have been reassessed to a higher threat category
  • Number of species that have been removed from the list of wildlife SAR because they are no longer at risk
  • % of listed species (% extirpated, % endangered, % threatened, % special concern) for which the recovery goals have been achieved (1, 2, 3, 4)
 DFO ADM O&HS, Regional DGs and SO, SARA

EC Associate ADM ESB, CEO

PC DG NP

Partners,  Aboriginal people and Stake-holders
SAR DatabaseDFO Director SARS

EC Director CSDP

PC Executive Director EI
2010- 2011

12 Those indicators that support risk management and assessment are presented in bold. The numbers found following indicators refer back to the list of risks outlined in Section 4.
13 Those indicators that related to the core departments’ workload are presented in bold and italic. These indicators demonstrate the volume of work in that area.
14 SAR database, managed by EC,contains information regarding the assessment, listing, recovery of species at risk and federal lands where they are known to occur. It is used for the management of the Program and also supplies data to the SARA Registry. Currently, information from the database is shared with DFO and PCupon request. However, negotiations are underway to provide full access toDFO and PC. The database also provides information on the general status of wildlife in Canada.
15 Wildlife Species website
16 Compliance promotion strategies and plans are internal departmental documents