Targeted control audit of CARM business readiness: Detailed audit results, recommendation and response
8.0 Audit findings
8.1 Managing key risks
18. According to the Treasury Board Secretariat (TBS), risk management is recognized as a core element of effective public administration. The effective management of risk contributes to improved decision-making, better allocation of resources and, ultimately, better results for Canadians.
19. CARM's project risk log identifies approximately 200 risks. Managing project risk is a shared responsibility between the agency and the Vendor. The audit assessed the agency's risk management processes for three CARM's R2 risks as they were determined to have the greatest potential to negatively impact the launch. Each risk was assessed using the Treasury Board's Guide to Integrated Risk Management. Of the three critical risks that the audit examined, we found that the agency did make an effort to mitigate each one once the risks were identified. However, risk management practices were not sufficient to ensure business readiness for CARM R2 full implementation in . Moving forward, opportunities exist to improve risk management practices.
8.2 CARM's legislative authority to operate
20. In , the CBSA had documented, via its risk registry, that there was a potential risk that legislative and regulatory authorities required for CARM would not be in place and therefore could delay the project. Specifically, authorities may be required to allow electronic payments from brokers and importers to be made to the CBSA dependent upon the proposed CARM solution. If required, the agency noted that this could take upwards of 36 months to be finalized.
21. The Statement of Work clearly identifies that CARM R2 must adhere to and ensure compliance with federal, legislative and TBS policies. The CARM IT design process began in 2018. It was also during this time that some officials noted that, while important amendments to the legal framework may not be essential to the launch they could, however, be implemented after CARM was launched. By 2019, once the design for the new system was well underway, the CARM program authority began determining what amendments would be required to existing legislation. In hindsight, this should have been identified earlier in the process during the early design stages.
22. Eventually, amendments which supported electronic payments and security deposits were drafted. In , however, the TBS informed the agency that part of its legislative amendments were being delayed. As a result, the CARM project team explored whether imported goods could be released prior to payment of duties without security. [Redacted]
Note: [Redacted] appears where sensitive information has been removed in accordance with the Access to Information Act and the Privacy Act.
23. Early identification of the need for legislative and regulatory authorities gave the agency time to react to this key risk. However, incorrectly assuming that legislative amendments were not required triggered a course of events which ultimately left the agency without the required authorities to meet the R2 launch. As of the writing of this report, the required amendment has yet to be fully completed.
24. Since 2018, the CARM Program Authority Unit had been actively discussing the legislative amendments with other key internal stakeholders as well as discussions with the United States Department of Homeland Security. In addition, governance and oversight bodies such as the CARM Project Board (CPB) and the Finance and Investment Management Committee (FIMC) were provided with updates on this risk. However, the audit's review of the CPB's records of decision for a 2-year period (from to ) could only confirm that three discussions took place on regulatory amendments. There was only one explicit mention regarding the potential delay these amendments could have on the CARM R2 launch date.
25. Given the potential implications of this risk materializing, it is concerning that CPB (which acts as a decision-making body for major CARM issues) did not discuss this risk more frequently. Communication between the project team and agency oversight bodies could be strengthened to ensure greater awareness of key risks to the CARM R2 planned launch.
26. From 2017 to 2019, the agency began working on Customs Act amendments needed to support CARM including e-payments and security that were being pursued through the Budget Implementation Act (BIA) and the Annual Regulatory Modernization Bill (ARMB 2020). Due in large part to the COVID-19 pandemic, the tabling of both bills was delayed by at least one year. As progress on the development of CARM R2 continued, it became apparent that the delay in the passing of the legislative amendments would have significant impact on the launch.
27. As the lead department responsible for proposed amendments contained in the BIA, the Department of Finance requested that the CBSA remove duplicate references in the BIA and ARMB 2020 in . The agency agreed and removed all duplicate provisions from the BIA. The BIA received Royal Assent on , but without the key e-payment and e-security amendments. Had the agency included these key authorities, the CBSA would have had what was required to operate CARM R2.
28. After the passage of the BIA, the ARMB 2020 was the legislative vehicle that contained the proposed Customs Act amendments to allow for regulatory amendments. However, the ARMB 2020 was not tabled prior to the end of the spring 2021 Parliamentary session and has yet to be tabled. As a result, the CBSA continues to be without the necessary legislative authorities to enable to required e-payment provisions. Although actions were taken to mitigate the risk of legislative authorities delaying the project, changes to legislation are dependent upon a parliamentary cycle and are not necessarily within the direct control of the agency.
29. Legislative and regulatory amendments, including those related to electronic payments, are necessary authorities for CARM to fully operate as intended. Without these authorities, the agency is not business ready for R2 launch as it will not have the legal authority to electronically mandate the collection of all revenues from importers and broker.
8.3 Trade chain partners onboarding and certification
30. When CARM becomes fully operational, all brokers and importers (trade chain partners, or TCPs) who bring goods into Canada will need to pay their duties and taxes through the CARM system. To do that, all TCPs need to register in the CARM Client Portal (also called TCP onboarding). Larger brokers and importers who desire to use their own reporting systems will need to do the following (which will collectively take 12 to 18 months):
- build their own IT capability to be able to use the CARM system
- have the CBSA certify the TCP as having the proper electronic systems to create, send, and receive data to and from the CARM system (also called TCP certification)
31. This onboarding and certification is crucial to the success of CARM as it ensures that all TCPs have the capability to transition to the new system and send the required payments to the CBSA. During the examination phase of the audit (end of ) the agency had very low onboarding numbers for importers, with only 6,165 out of an estimated total of 240,000 on-boarded to the CARM portal.
32. Low onboarding and certification was identified as a high risk as early as . Later in , the agency also acknowledged that the TCP community may not be in a position to prioritize CARM-related work due to pandemic pressures. Although this risk had clearly been identified, no specific mitigating actions had been decided upon. Low TCP onboarding rates also increases the risk that TCPs may not be ready to adopt the system; thereby impacting the agency's ability to fully realize CARM's full economic benefits.
33. A review of records of decisions, along with interviews, indicate that oversight bodies including the CARM Project Board and Branch Management Committee received regular briefings on the status of TCP onboarding and certification.
34. The agency was not regularly tracking and reviewing the TCP onboarding rates. This impacted the agency's ability to accurately determine the extent to which TCP onboarding and certification was occurring. The agency began monitoring TCP Onboarding rates as of through biweekly and CARM status reports when it was identified by the audit as a gap.
35. Records of decisions were reviewed for the TCP working group (a working group dedicated to client onboarding) and noted that the agency developed a communication and onboarding strategy. This strategy was geared toward larger TCPs and resulted in the registration of the top 30 brokers, who account for more than 80% of all broker volume and value as well as 821 of the top 3,000 importers who account for more than 80% of importer value. However, it has left a gap for smaller and medium-sized businesses, with less than 2% of these TCPs having on-boarded to date. It is unclear how these risks will be mitigated prior to the launch.
36. Although the agency is currently tracking TCP onboarding rates, there is still no capability to track and/or determine which TCPs are certified to receive and send data from the CARM system. In the absence of adequate TCP onboarding and certification, the agency runs a risk that the importers and brokers in the TCP community will not be ready to adopt the system at the time of its launch thereby potentially negatively impacting the flow of goods at the border.
8.4 Integration with the enhanced Tariff Risk Assessment Service
37. Tariff Risk Assessment Service (TRAS) is a legacy IT service that is used to receive import information from TCPs to assess tariffs (duties/taxes). As part of CARM, TRAS was being updated to an enhanced TRAS (TRAS+) to align with CARM cloud technology infrastructure. This update was initially intended to be completed as part of the R1 CARM release. However, systems integration testing found significant issues with TRAS+ which could not be fixed in time to meet the initial R1 deadline. As a result, the TRAS+ update was added to the CARM R2 launch. Without TRAS+, the GC cannot properly assess revenue it is owed in duties and taxes.
38. The audit found that this issue was identified in the CARM status report. In response to the issue, the agency proposed shifting the TRAS+ release from R1 to R2. This received approval in . To resolve the system issues, the vendor was tasked with completing fixes to the system. In addition, a working group was established consisting of representatives from the vendor and the agency responsible for conducting planned approval checkpoints to ensure that needed requirements, along with the final mitigation measures, were agreed upon. Given the complexities of CARM R2, the added system work impacted the agency's workload, thereby increasing the risk of a delay. However, actions have been taken to resolve issues with TRAS+.
39. Systems integration testing in R1 identified that TRAS+ had significant issues to be addressed. We heard conflicting accounts from the CARM team regarding the underlying reasons why TRAS+ was not completed on time. This related to whether or not key system requirements (to ensure that TRAS+ functioned properly) were provided to the vendor. Given that TRAS+ was not completed in time for CARM R1 , it put additional stress on an already compressed CARM R2 work schedule.
40. The agency continued to monitor TRAS+'s changing environment through the working group discussions, updates and action items, as well as biweekly CARM status reports which contained progress on TRAS+. These reports are also shared with Senior CARM leadership. TRAS+ completion deadlines for R2 originally scheduled for , and subsequently , were missed. According to an CARM status report, TRAS+ is still not finalized with no completion date identified.
41. In the absence of a properly functioning TRAS+, the agency is at risk of not being able to assess duties and taxes owed to the Crown by importers and exporters, thereby impacting the agency's ability to operate the CARM system and realize all expected benefits. Although the TRAS+ risk materialized, the agency has taken actions to address and mitigate this risk. It is critical that continued monitoring and communication occur in order to allow for the successful completion of TRAS+ prior to launching.
8.5 Internal and external readiness activities
42. The audit focused on assessing key activities that could impact the agency's ability to be business ready by .
8.5.1 Internal business readiness activities
8.5.1.1 Operational impact assessments
43. Operational impact assessments (OIAs) identify branch level business readiness gaps and remediation activities to maximize the likelihood that internal stakeholders are business ready for CARM R2. A review of the OIA tracking spreadsheet found that all thirty-four R2 OIA assessments were scheduled to be completed by and were at various stages of completion. However, as of , no R2 OIAs had been fully completed. Given that our audit work ended in , we were unable to conclude whether or not the R2 OIA assessments will be ready as scheduled for .
44. As R1 underwent its own OIA process, the audit assessed whether lessons learned from the R1 assessments were incorporated into R2 assessments. Lessons learned from the R1 OIAs were presented at CBRIB in . Through interviews management confirmed that R1 OIA lessons learned were to be applied to the R2 process, including starting the OIA process sooner and giving special consideration to monitoring workload increases. Although the R2 OIAs were in their early stages, we saw evidence of this occurring. Engagement with business areas (HQ and Regional Corporate Services Divisions) was occurring regularly between and in an effort to identify branch level readiness gaps.
8.5.1.2 CARM Client Support Helpdesk
45. The CARM Client Support Helpdesk (CCSH) supports client onboarding to the CARM Portal and provides contacts for internal and external clients to resolve issues. However, while the CCSH supports client onboarding, it is the agency's stakeholder engagement activities with TCPs that drives portal registration and awareness. The CCSH was operationalized on – the day before the CARM R1 go-live date. Prior to this date, CCSH officer roles and responsibilities had been defined and shared in an Officers Handbook and training was given to team members.
46. The audit reviewed the CCSH Training Plan to assess the extent of training and found that there were 90+ courses and readings in the CCSH curriculum. Further to this, a training schedule confirmed that CCSH officer training was delivered between and ; management validated that CCSH officers completed training prior to R1 go-live.
47. To assess CCSH operational activity, the audit reviewed operational activity reports, which are produced internally by the CCSH and shared at weekly team meetings including:
- statistics on call volumes
- identification of challenges and issues
- lessons learned with steps taken to address them
48. Standing up and operationalizing the CCSH has helped manage internal and external client issues as they onboard and navigate the system. The CCSH represents a positive activity in support of business readiness.
8.5.1.3 CARM Business Readiness Implementation Board
49. The mandate of the CARM Business Readiness Implementation Board (CBRIB) is to provide oversight of progress and accountability toward CARM business readiness and implementation. The CBRIB terms of reference state that the board is responsible for overseeing the agency's preparedness and execution of business readiness activities that the CBSA needs to execute and prepare for each CARM release.
50. CBRIB plays a vital role in CARM project governance because it complements the CARM Project Board (CPB), which provides governance for the CARM project within the CBSA's Project Management Framework (PMF). Both CPB and CBRIB report to the Executive Committee (EC), the CBSA's senior management decision-making forum.
51. The audit reviewed all dashboards and Records of Decision (RODs) from to to determine the extent to which CBRIB was providing effective oversight for the agency's preparedness for CARM R2. While we found CBRIB provided effective oversight of the business readiness activities tabled at committee, we also noted a gap with regard to CBRIB's reporting on risk management. Specifically, only the ROD from CBRIB referenced the need to monitor legislative and regulatory authority; therefore it is unclear how the CBRIB supported EC with effective oversight on a major risk impacting the agency's ability to deliver CARM R2 in .
52. The CBRIB's oversight of operational business readiness helps ensures key risks are properly managed. The absence of effective upward reporting to senior management governance committees on key risks increases the likelihood that senior decision-makers are not provided with timely advice and/or feedback on CARM business readiness in order to manage risks appropriately.
8.5.1.4 Integrated project plan
53. The integrated project plan (IPP) is the main CARM project plan. It is reviewed to ensure that any changes in timelines are reflected in an updated IPP. Keeping the IPP up to date is important for tracking CARM R2 deliverables that may impact the ability of a business area to complete an activity required for business readiness. Against this backdrop, the audit reviewed the IPP to ensure changes in timelines were reflected in an updated IPP.
54. Evidence showed that the IPP was indeed updated to reflect changes in timelines, specifically following two important requests for change in and . However, there is an opportunity to utilize the IPP as a more effective oversight tool; specifically as an early warning mechanism for deliverables which are in jeopardy of missing their completion date and subsequently creating risks to business readiness. For example, the ARMB legislative amendments were marked "late" in the IPP (expected completion date was ). However, the CBRIB had not reviewed this risk in the calendar year up until that point. Therefore, if the CBRIB were to review the IPP as a standing agenda item, this could enable the CARM project to proactively address business readiness risks before they materialize.
55. Internal activities have demonstrated good progress toward being business ready; however CBRIB and the IPP are examples of opportunities where the agency can more effectively support CARM.
8.5.2 External business readiness activities
8.5.2.1 Trade chain partner survey
56. The trade chain partner survey seeks to provide the CARM project team with insights on the level of TCP change readiness in order to tailor material (i.e. job aids) and communications for TCPs for R2. An initial engagement survey for R2 was completed in , with a second survey scheduled for . The first survey was effective in reaching out to the external community by asking questions on preferred methods of communication and exploring future change enablement activities. Management collected valuable information on what engagement activities most impact TCP's change readiness status. This will help to inform future communications and onboarding products. For example, continuing the use of email (the mailbox) and website updates and provide more action-focused products (i.e. checklists and worksheets) on what is needed to be ready for CARM R2.
57. The CBSA may wish to leverage information collected from the TCP Working Group to refine the survey based on the known challenges faced within the TCP community, thus demonstrating to TCPs that the agency is "listening" to their concerns, which may positively impact buy-in for CARM R2. For the upcoming survey scheduled for , sufficient time should be dedicated to identify gaps and effectively implement corrective action(s) before the launch of CARM R2.
8.5.2.2 Stakeholder engagement mailbox
58. The stakeholder engagement mailbox is used for internal/external communications, scheduling engagement, and proactively communicating with TCPs. It was established in 2018 and utilizes a documented process to respond to TCP messages, with subject matter experts being engaged as needed.
59. By reviewing a sample of emails from the Stakeholder Engagement Mailbox, the audit sought to determine its effectiveness by analyzing the following:
- how information is considered
- the timeliness of responses
- how issues are addressed
60. We found that the mailbox has logged more than 1,000 questions from TCPs since its inception. External communications through the mailbox are conducted in a variety of ways, including questions and answers being posted biweekly on a Google documents site. This provides TCPs with solutions to issues they may be encountering. The mailbox was ranked in the TCP Engagement Survey as the preferred communication method for TCP's to engage with the agency.
61. As the agency moves toward launching CARM R2, ensuring an effective communication channel with TCP's is paramount to a successful launch. The mailbox has been a positive step for engaging feedback from the TCP community. We encourage the agency to continue to utilize it.
8.5.3 CARM training
62. The CARM Training Unit (CTU), Change Enablement Division, has a vital role to play for ensuring that CARM training delivery plans and engagement materials are developed and communicated. The vendor is responsible for all training delivery. While the audit did not assess contractor performance as it relates to training, it did conduct a review of the CBSA's role with regard to training given that the agency provides final approval for all training materials and delivery methods.
63. Training for CARM R1 provided a series of good practices that are planned to be applied to R2. Specifically, given the constraints associated with the global pandemic, the agency approved training delivery methods that included online virtual instructor-led sessions, self-paced learning, the SAP Enable Now platform and the use of YouTube. Overall, the feedback from the R1 sessions was that the agency should apply these lessons learned in order to maximize training availability for R2. The agency is also in a good position to pivot and or/adjust should COVID-19 restrictions be lifted, given that CTU has training facilities at Rigaud that can be utilized for CARM R2 training.
64. As of , the R2 training strategy has been developed and the CTU was on track to have final training materials prepared for . Although the R2 training strategy itself is complete, it should be noted that training materials cannot be produced until after user acceptance testing (UAT) is finalized. As a result of CARM R2's 8-week solution build delay (as of ) the CTU issued a request for change to add contractors for translation services and adjust timelines. This is a positive step to address translation capacity issues which negatively impacted the delivery of translated training materials for R1. However, since training timelines are heavily dependent upon the solution build, there may be insufficient time allocated for training for a R2 should the build be delayed further. Continued solution build delays have compressed training timelines, which could have significant negative impacts on the agency's ability to be business ready for R2.
65. The CBSA is meeting its responsibilities for training and has established delivery methods suitable for pandemic restraints. However, training plans and materials are being impacted by solution build delays and may compress training timelines.
8.6 Information technology systems integration and security
66. The CBSA is responsible for ensuring that the CARM system meets requirements outlined within the statement of work (SOW). The SOW contains a list of deliverables agreed upon between the vendor and the CBSA. One of the primary ways to ensure the system is meeting requirements is through UAT. UAT serves as one of the last types of tests performed before the system goes live. Once the CBSA signs-off on UAT, it indicates that the agency believes the system meets its requirements and can be effectively implemented. In order to determine the agency's progress on key operational initiatives for business readiness for an R2 launch (), the audit assessed key IT systems integration and IT security.
8.6.1 IT systems integration
67. IT Systems Integration refers to the process of developing the CARM IT solution and integrating it with the existing CBSA infrastructure and legacy systems. As outlined in the statement of work, the vendor is responsible for all system integration. However, the CBSA is ultimately responsible for ensuring that the system being developed meets its requirements. This is primarily done through UAT. UAT relates to testing the software to confirm that the software is functioning correctly as per the requirements.
68. For CARM R1, while 8 weeks were planned for UAT, it required 16 weeks to complete. Following systems integration testing, a proposal was made to stand up a Tiger Team to address the integration issues that were uncovered. Instead, a consultant-led assessment of CARM interfaces for R2 was conducted by the CARM team from to to proactively mitigate the risks. This assessment's findings were presented at a CARM Leadership Meeting in late . At the meeting, a number of key issues were raised including TRAS, as well as CARM and CRA integrations.
69. Both the CARM Status report and Gartner health check identified risks related to system integration but the limited time for UAT testing remained unaddressed. Currently 7 weeks of UAT is planned for R2; however agency officials have indicated that 3 to 4 months is required. UAT is one of the last types of tests performed before go-live and once approved indicates that the system meets its requirements and can be implemented. Solution build delays of 8 weeks, as of , have contributed to system integration testing not beginning as scheduled in the IPP.
70. When R2 is launched, some legacy systems will no longer be used, which is different from R0 and R1, where legacy systems continued to operate. Therefore, effective oversight and testing are critical for CARM R2 as it is significantly more complex and riskier than R1. As of , there was no contingency plan for CARM R2 launch if system requirements were not met.
71. Significant delays to the solution build have impacted critical integration activities that need to be performed, including addressing interface issues and performing UAT testing. Without adequate time allocated to effectively complete this work, there is an increased risk that the system will not meet all of its requirements. This could lead to the system not working as intended, posing legal and reputational risks to the agency.
8.6.2 IT security
72. The IT SA&A process ensures that appropriate security assessments for both the GC and Cloud Service Provider (CSP) will be completed prior to CARM R2 go-live. The CBSA has aligned its requirements and guidelines from Central Agencies to its Service Lifecycle Management Framework (SLMF) through the Security Management Control Method (SMCM).
73. The deliverables for CARM R2's SA&A include 4 sequential steps:
- Critical Security Assessment Report (CSAR): The R2 CSARs for both the GC and the CSP were completed and presented to CBSA senior management in .
Although planned in the IPP, the following steps have not yet been completed and could not be assessed:
- Final security assessment report (FSAR): This report is critical in identifying any remaining security risks.
- Security management action plan (SMAP): This action plan addresses potential security risks identified in FSAR.
- Authority to operate: This is the final step, to launch and operate CARM R2.
74. Due to significant delays in the CARM system build for R2, the start date for the FSAR has been delayed. According to the most up-to-date status report, the FSAR is scheduled to begin in ; only three months before R2 is scheduled to launch in . The FSAR and SMAP took approximately 7 months to complete for R1. Therefore, sufficient time must be allocated for the FSAR and SMAPS for R2 in order to ensure that all IT issues have been addressed before launching the system.
75. The SA&A process is a critical component that must be in place prior to launching R2. However, in the absence of a completed solution build, the agency cannot fully carry out critical activities to complete its SA&A process and launch the system. The agency's SA&A process aligns with security requirements defined by central agencies. However, the agency should ensure that they allocate sufficient time for the FSAR and the SMAP to be completed; in order to ensure that all IT issues have been addressed before the agency launches the system.
Recommendation
The Vice-President of Commercial Trade Branch should revise the current set of project updates to senior management and governance committees to better highlight key risks, mitigation strategies, timelines and budget considerations in order to support timely, informed and strategic decision-making for all high-risk areas.
Management response
The Vice-President of the Commercial and Trade Branch will provide regular, targeted and strategic updates to senior management through the use of dashboards, reports or briefings, and will work toward formal alignment amongst business delivery partners to implement effective risk mitigation and focus on CARM readiness and delivery of R2.
CARM project resources continue to work closely with business partners to further the understanding of the CARM solution; support the identification of new/changed business processes to operationalize CARM; ensure business readiness plans are reviewed against targeted R2 timelines; and, hold individual sessions with business lead representatives to understand where/how the project can provide support to further readiness for the final release.
Overall completion date:
- Date modified: