Failure mode and effects analysis for errors detected during pretreatment physics plan and chart review in external beam radiotherapy

Introduction

Quality assurance (QA) plays a critical role in patient safety during radiation therapy. Among various QA checks, pretreatment physics plan and chart review (PTPCR) is sensitive at detecting errors before beam delivery (1). Therefore, it is important to ensure the quality and effectiveness of PTPCR to provide accurate and safe treatment. An effective PTPCR can detect errors that can cause some patient safety issues or reduce the efficiency of the clinical workflow due to error clearing. Lack et al. reported that data errors (e.g., wrong patient identification entered, incorrect CT-density curve applied, etc.) occurring during treatment planning directly affect the quality and generation of the plans, delaying the initiation of treatment because of the time costs incurred from plan modification (2). However, if a small percentage of errors still cannot be detected by the PTPCR, indicating the need to perform certain interventions for improving the PTPCR (1).

Although PTPCR is one of the main medical physicist’s professional duties, no agreed standard checklist for PTPCR exists at present. Moreover, the ability of physicists to review plans varies according to individual training and experience, and no standard approach to quantifying this ability is available. A survey in North America indicated that the observing and practicing methods are commonly used when training resident physicists to conduct the PTPCR (3). Therefore, some residency training courses related to the PTPCR have been promoted in North America. Another study used simulated errors in treatment plans to determine the ability of physicists to detect errors and the effectiveness of PTPCR; such plans with simulated errors could also assist with the training and education related to PTPCR (4).

PTPCR has been an important task to ensure the safety of radiotherapy; however, if a small percentage of errors still cannot be detected by the PTPCR, indicating the need to perform certain interventions for improving the PTPCR (1). Gopan et al. indicated that further improvements are required in clinical PTPCR and recommended the use of automation and standardization for PTPCR (5). In addition, several researchers have begun to automate the PTPCR through electronic forms to improve the detection of errors and reduce treatment delays (6-8). However, despite the availability of automated electronic review methods, an effective review form is still indispensable for some items that cannot be reviewed by automated processing.

Failure mode and effects analysis (FMEA) is a widely used tool for the methodical analysis of workflow and the detection of factors that affect safety. FMEA has been applied to improve the treatment process, minimize the errors during treatment, and increase the safety of various radiotherapy techniques, including stereotactic body radiation therapy, proton therapy, intraoperative radiation therapy, brachytherapy, tomotherapy, and radiosurgery (9-15). The American Association of Physicists in Medicine’s (AAPM) Task Group (TG)-275 identifies key high-risk failure modes (FMs) based on the FMEA results to provide strategies for the effective use of PTPCR in radiation therapy (16).

We would like to use a standard checklist to perform PTPCR in our department for different radiotherapy techniques: volumetric modulated arc therapy (VMAT), intensity-modulated radiotherapy (IMRT), tomotherapy, 3-dimensional (3D) conformal radiotherapy, hand-calculation electron radiotherapy (ERT) and 2D radiotherapy. In this study, we aim to use FMEA to evaluate the errors detected during the PTPCR and to summarize and classify the types of errors. In addition, we monitor the effectiveness of the PTPCR by analyzing the number of errors detected by radiation therapists during the plan acceptance check, which included three major review process steps: (I) image parameters, (II) record and verify (R&V) system parameters, and (III) chart document. We report our experience in applying FMEA in the PTPCR for other practitioners, and the results can be considered as a reference for PTPCR checklist. We present the following article in accordance with the SQUIRE reporting checklist (available at https://tro.amegroups.com/article/view/10.21037/tro-21-38/rc).

Methods

Department data environment

The radiation therapy facilities in our department included a 16-slice Brilliance Big Bore CT simulator (Philips Medical Systems, Cleveland, OH, USA), 2 linear accelerators (Elekta, Stockholm, Sweden), and a Hi-Art tomotherapy system (Version 4.1.2, TomoTherapy Inc., Madison, WI, USA), a R&V system database (MOSAIQ, Version 1.60, Elekta, Stockholm, Sweden), and a centralized treatment planning system (TPS; Pinnacle, Version 14.0, Philips, Fitchburg, WI, USA).

Treatment planning

In 2019, 1406 treatment plans were generated using VMAT (52.3%), IMRT (30%), helical tomotherapy (HT) (13%), 3D conformal radiotherapy (1.9%), direct tomotherapy (DT) (1.6%), hand-calculation ERT (0.9%) and 2D radiotherapy (0.3%). For curative radiation treatment, the goals for target coverage were as followed 97% of the prescribed dose (PD) should cover at least 97% of the planning target volume (PTV) (PTV V97%PD ≥97%) for abdominal and pelvic cancers, and PTV V95%PD ≥95% for breast, chest, head and neck cancers, and limbs. All plans were deemed clinically acceptable by the radiation oncologists based on the trade-off between the target coverage and the doses to organs at risk according to the guidelines of Quantitative Analysis of Normal Tissue Effects in the Clinic (QUANTEC). Doses to organs at risk and target coverage were used to evaluate the plan quality.

PTPCR

Five board-certified physicists in our department with an average of 9 (range: 5−12) years of experience used a single checklist to perform the PTPCR. The workflow of PTPCR in this study included the review of the treatment plan (TPS check), relevant chart documents (chart check), and R&V system (R&V check). Every treatment plan was sent for PTPCR and the errors detected should be corrected before treatment. All the errors detected were recorded, classified, and tabulated for further analysis.

FMEA

The FMEA described in AAPM TG-100 was used to assess and quantify risk (17). This method involves compiling FMs (i.e., the ways in which something could go wrong) with related causes and assigning scores to each mode. The scoring system for FMEA included three parameters, occurrence (O), severity (S), and detectability (D) with specified qualitative score from 1 to 10. First, a process map (Figure 1) including a major process tree and subprocess steps was generated based on the guidelines in AAPM TG-100 (17). Errors detected during the PTPCR identified in FMs by the physicists according to the error type (Table 1). Second, the five physicists in our department used the scoring system (Table 2) assigned numeric values to O, S and D for each FM. In this study, we facilitated the qualitative descriptions of occurrence, severity, and detectability to be understood by modifying the scoring system based on TG-100 (17) and several previous publications (14,18,19), which will help physicists scoring. Table 1 presents a local database of error types and error rates that were used for occurrence ranking in this study. The five physicists discussed the values of O, S and D to minimize the randomness in scoring. Finally, the risk priority number (RPN) (the product of O, S, and D) and criticality (the product of S and D) were used for risk assessment in the FMEA. FMs with RPN scores >100 were high risk defined in TG-275 (16). High RPNs indicate more critical defect in process (17).

Figure 1 PTPCR process map. PTPCR, pretreatment physics plan and chart review; TPS, treatment planning system; R&V, record and verify; CT, computed tomography; OAR, organ at risk; MU, monitor unit; DRR, digitally reconstructed radiograph; CBCT, cone-beam computed tomography; SSD, source-to-surface distance.

Assessment of quality and efficiency

Descriptive statistics include single, multiple and overall error rate, which were calculated to assess the quality and efficiency of the planning. Single or multiple error rate was defined the number of plans with single or multiple detected errors divide the total number of plans. The overall error rate obtained from the summation of single error rate and multiple error rate. The PTPCR compliance rate, calculated as the number of plans without errors detected during the plan acceptance check/the total number of plans, was used to quantify the effectiveness of the PTPCR (20).

Statistical analysis

All analyses were conducted using Excel (Microsoft, Redmond, WA, USA). There was no any statistical analysis used in this study because it was the report of the clinical experience in our institution.

Ethical statement

The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the Taipei Medical University Joint Institutional Review Board (No. N201809006) and individual consent for this analysis was waived.

Results

Process mapping

Figure 1 shows the process map of the workflow of PTPCR checks at our department. The process map includes five major review process steps: (I) planning parameters, (II) plan quality, (III) image parameters, (IV) MOSAIQ management, and (V) chart document. Each major process step includes several subprocesses, consisting of a total of 35 subprocess steps.

FMEA

Figure 2 shows the distribution of O, S, and D values vs. RPN scores (ranging from 19.2 to 155.5) for the 35 FMs identified in this study. Most FMs with RPN scores <100 had low S and D values (≤4), but high O value (≥6). Most FMs with RPN scores >100 had low D (<4), mid-range O (4–6), and high S values (>6). All values of O, S, and D ranged from 4 to 9, 2 to 10, and 2 to 8, respectively. The subprocess step with the highest O value (9) was “isocenter shifts” (major process: chart document). The subprocess steps with the highest S value (10) were “primary image set” and “hand monitor unit (MU) calculation” (major process: image and planning parameters). The subprocess step with the highest D value (8) was “DRR/CBCT image set” (major process: image set).

Figure 2 Distribution of O, S, and D values versus RPN score for FMs identified in this study. The overlaid FMs are expressed as one symbol. O, occurrence; S, severity; D, detectability; RPN, risk priority number; FMs, failure modes.

Figure 3 shows the number of FMs, mean RPN, highest RPN, and mean criticality for each major process step of our PTPCR. The chart document had the highest number of FMs and occurrence rate (89/44.3%), followed by MOSAIQ management (52/25.9%), and the planning parameters ranked third (49/23.9%). The top-ranked FMs for these 3 major process steps were couch-density correction, isocenter shifts, and field-image definition, as shown in Table 1. The subprocess step with the highest RPN scores (155.5) was “skin flash” (major process: planning parameters). The major process steps with a mean criticality >30 were image parameters (31.0) and plan quality (30.2). The major process step with the highest mean RPN scores was image parameters (121.6).

Figure 3 Distribution of the number of FMs, mean RPN, highest RPN, and mean criticality for each major process of our PTPCR. FMs, failure modes; RPN, risk priority number; PTPCR, pretreatment physics plan and chart review.

Ten FMs with RPN scores >100 are listed in Table 3, which occurred in the major process steps of chart document, image parameters, and planning parameters. The subprocess step with the highest criticality value was “isocenter shifts” (major process: chart document).

Assessment of quality and efficiency

A total of 168 plans with 201 errors were detected from 1406 PTPCRs in 2019. Table 1 shows the number and occurrence rate of each error/FM. Among these plans, 150 plans had a single error and 18 plans had multiple errors. The mean rates of single, multiple, and overall errors were 10.6%, 1.3%, and 11.9% respectively. After the PTPCR, 140 errors from 136 plans were detected by the radiation therapists during the plan acceptance check which included three major review process steps: (I) image parameters, (II) MOSAIQ parameters, and (III) chart document. The common sources of error detected after the PTPCR were DRR/CBCT image (43.6%), plan/field name (15%), plan status (8.6%), radiation prescription (8.6%) in MOSAIQ management, and isocenter shifts (5.7%) in chart document. The PTPCR compliance rate was 90.3%.

Discussion

The FMEA provides a systematic and effective method for evaluating errors in TP and PTPCR. Our PTPCR results helped us to summarize the types of errors and the occurrence rate of these errors. Our department did not implement paperless charts (electronic charts) until May 2020; paper charts were still used in 2019, which contained TP-related information including isocenter information, MUs, source-surface distance, etc. Errors in this handwritten information may lead to delays in treatment or dosimetric errors. For example, incorrectly written isocenter information in the chart could lead to radiation therapists taking the verification films according to the wrong information. Consequently, more time would be required to adjust the displacement and retake the verification films. Similarly, missing information regarding the bolus required could lead to a wrong dose distribution. Electronic charts were implemented in May 2020 in our department, which obviated the requirement to transcribe the TP information in the chart. All the treatment information accessible to the radiation therapists in the electronic chart system (ECS) can be acquired from the treatment plan PDF by parsing the PDF file and the R&V database. However, it is important to ensure the accuracy of data transfer between different systems such as the TPS, R&V system, treatment control system, and ECS. Moreover, the efficiency of the radiation therapists in accessing the treatment information is also important. Owing to the high occurrence rate of chart document errors in our department and according to the TG-275 recommendations, we have been attempting to reduce the manual transcription of treatment information and data entry in the R&V system.

The first step in FMEA is to construct a process map, which can help the physicists in our department arrive at a consensus on the PTPCR process. The process map structure allowed the physicists to become more familiar with what needs to be checked in each major PTPCR process step without missing the details on checks of important steps. With regard to the five major process steps of the PTPCR in our department, physicists required some time to cross-check each parameter between the TPS, R&V system, and chart using a single checklist. For more complex treatment plans, physicists typically required more time to complete the PTPCR, increasing the possibility of interruption, which may affect PTPCR quality. In our experience, performance of more than two PTPCRs at a time could reduce the possibility of detecting errors during the PTPCR. To avoid a longer review time, our physicists performed one PTPCR at a time and performed at most two PTPCRs in a row. To facilitate the process, for complex or special plans, additional planning details and information should be provided to the reviewing physicists before a PTPCR is performed. Some problems encountered during the PTPCR in this study need to be addressed in the future, such as a distraction-free work environment and adequate time for a PTPCR. In our department, all new PTPCRs are not assigned to the same one physicist per day but to the physicist who is available when plans are generated and approved by the physician. According to the TG-275 recommendations, the PTPCR should be performed by a certified medical physicist or the process should be conducted under their supervision (16).

The scoring step in FMEA involved the five physicists in our department. The scoring process in FMEA is somewhat subjective, although we discussed the scoring table before assigning values to reduce misunderstandings and scoring bias. Our results showed that 10 FMs with a RPN value greater than 100 (Table 3), and 7 of these FMs (except for couch-density correction, heterogeneity correction, and CT-density curve) are also reported in TG-275 Tab. S1.A.i. listing high-risk FMs (16). Based on our FMEA results, some interventions have been introduced into our clinical workflow. The ECS implemented in our department can avoid the incorrect handwritten isocenter shifts in charts, which had the highest criticality value. The omission of DRR/CBCT image set transfer from TPS to the on-board imaging system had the highest detectability score and was the major source of errors detected after the PTPCR by radiation therapists. Therefore, we strictly request that planners transfer DRR/CBCT images before a PTPCR. In our ECS, we have set a “checkpoint” in the workflow, which means that the planner must complete the self-check on this item to continue the review process. A script has been written in the TPS for couch-density correction so that the planner can run the script before optimization. With regard to the MOSAIQ R&V system, which was the second major source of errors detected after PTPCR, another script has been written in the TPS to help correctly export the field parameters and DRRs to the R&V system. However, some information still needs to be manually entered in our current version of the R&V system, such as the association of treatment fields with images and doses for each treatment field. Again, for these items, we established “checkpoints” in the ECS for self-check by the planner. In 2020, 1,311 PTPCRs were performed. The interventions mentioned above reduced the number of three most common errors (shown in Table 1) from 40 to 0 for isocenter shifts error, 15 to 7 for couch-density error, and 27 to 19 for image definition error. Moreover, the number of errors detected after the PTPCR in two process steps, DRR/CBCT image and MOSAIQ management which accounted for over 80% of errors detected after PTPCR in 2019, was reduced by the interventions from 119 to 84. To improve the effectiveness of the PTPCR and the efficiency of the clinical workflow, we will continue optimizing our ECS, and introducing the interventions based on the FMEA results.

In this study, we obtained the TP error rates (single, multiple, and overall error rates) and the PTPCR compliance rate. These results can help us realize the quality of the TP and the effectiveness of the PTPCR. After the improvement, we will continue to monitor the quality of our review work. Automated plan reviews have been proposed in recent years to facilitate human plan reviews (6-8). However, we believe that a human plan review is still required, especially for complex treatment plans, such as total body irradiation and stereotactic radiosurgery. In addition, physicists can gain a better understanding of different planning techniques through PTPCR. Schubert et al. reported that the pretreatment physics plan review is not simply paperwork; physicists should focus on the problem solving (3). Therefore, we believe that physicists can gain useful and valuable experience from this review work. Depending on the personnel resources of radiation oncology departments in different regions, small departments can still use a simple checklist to assist in the pretreatment physics plan review, and TG-275 has provided some recommendations in this regard (16).

A limitation of this study is that the physicists would occasionally miss to record the errors detected in their PTPCRs, which could lead to a statistical error in the number of FMs. However, the results of this study may still have value if they are considered as a reference by other institutions for designing their own review checklists in clinical practice.

According to the PTPCR results, we can summarize the types of errors and determine the occurrence rate of these errors. The FMEA serves as a systematic and useful method of evaluating the errors detected in a PTPCR. Determining the PTPCR compliance rate could help us understand the effectiveness of our PTPCR processing. On the basis of the FMEA results and PTPCR compliance rate, we performed some interventions to improve the effectiveness of the PTPCR to ensure the safety of treatment and the efficiency of the clinical workflow. We will continue to monitor the FMEA results and PTPCR compliance rates using the corresponding rates from this study as a baseline.

Acknowledgments

The authors thank all the medical physicists, radiation oncologists and therapists from Department of Radiation Oncology of Shuang Ho Hospital, Taipei Medical University for their generous encouragement.

Funding: This work was supported by Taipei Medical University—Shuang-Ho Hospital (grant number: 108HCP-17).

Footnote

Reporting Checklist: The authors have completed the SQUIRE reporting checklist. Available at https://tro.amegroups.com/article/view/10.21037/tro-21-38/rc

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://tro.amegroups.com/article/view/10.21037/tro-21-38/coif). ACS serves as an unpaid Associate Editor-in-Chief of Therapeutic Radiology and Oncology. The other authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the Taipei Medical University Joint Institutional Review Board (No. N201809006) and individual consent for this analysis was waived.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Ford EC, Terezakis S, Souranis A, et al. Quality control quantification (QCQ): a tool to measure the value of quality control checks in radiation oncology. Int J Radiat Oncol Biol Phys 2012;84:e263-9. [Crossref] [PubMed]
2. Lack D, Liang J, Benedetti L, et al. Early detection of potential errors during patient treatment planning. J Appl Clin Med Phys 2018;19:724-32. [Crossref] [PubMed]
3. Schubert LK, Hendrickson K, Miften M, et al. The Current State of Physics Plan Review Training in Medical Physics Residency Programs in North America. Pract Radiat Oncol 2020;10:e166-72. [Crossref] [PubMed]
4. Gopan O, Smith WP, Chvetsov A, et al. Utilizing simulated errors in radiotherapy plans to quantify the effectiveness of the physics plan review. Med Phys 2018;45:5359-65. [Crossref] [PubMed]
5. Gopan O, Zeng J, Novak A, et al. The effectiveness of pretreatment physics plan review for detecting errors in radiation therapy. Med Phys 2016;43:5181. [Crossref] [PubMed]
6. Covington EL, Chen X, Younge KC, et al. Improving treatment plan evaluation with automation. J Appl Clin Med Phys 2016;17:16-31. [Crossref] [PubMed]
7. Halabi T, Lu HM, Bernard DA, et al. Automated survey of 8000 plan checks at eight facilities. Med Phys 2016;43:4966. [Crossref] [PubMed]
8. Halabi T, Lu HM. Automating checks of plan check automation. J Appl Clin Med Phys 2014;15:4889. [Crossref] [PubMed]
9. Perks JR, Stanic S, Stern RL, et al. Failure mode and effect analysis for delivery of lung stereotactic body radiation therapy. Int J Radiat Oncol Biol Phys 2012;83:1324-9. [Crossref] [PubMed]
10. Veronese I, De Martin E, Martinotti AS, et al. Multi-institutional application of Failure Mode and Effects Analysis (FMEA) to CyberKnife Stereotactic Body Radiation Therapy (SBRT). Radiat Oncol 2015;10:132. [Crossref] [PubMed]
11. Zheng Y, Johnson R, Larson G. Minimizing treatment planning errors in proton therapy using failure mode and effects analysis. Med Phys 2016;43:2904-10. [Crossref] [PubMed]
12. Rash D, Hoffman D, Manger R, et al. Risk analysis of electronic intraoperative radiation therapy for breast cancer. Brachytherapy 2019;18:271-6. [Crossref] [PubMed]
13. Mayadev J, Dieterich S, Harse R, et al. A failure modes and effects analysis study for gynecologic high-dose-rate brachytherapy. Brachytherapy 2015;14:866-75. [Crossref] [PubMed]
14. Shen J, Wang X, Deng D, et al. Evaluation and improvement the safety of total marrow irradiation with helical tomotherapy using repeat failure mode and effects analysis. Radiat Oncol 2019;14:238. [Crossref] [PubMed]
15. Xu AY, Bhatnagar J, Bednarz G, et al. Failure modes and effects analysis (FMEA) for Gamma Knife radiosurgery. J Appl Clin Med Phys 2017;18:152-68. [Crossref] [PubMed]
16. Ford E, Conroy L, Dong L, et al. Strategies for effective physics plan and chart review in radiation therapy: Report of AAPM Task Group 275. Med Phys 2020;47:e236-72. [Crossref] [PubMed]
17. Huq MS, Fraass BA, Dunscombe PB, et al. The report of Task Group 100 of the AAPM: Application of risk analysis methods to radiation therapy quality management. Med Phys 2016;43:4209. [Crossref] [PubMed]
18. Frewen H, Brown E, Jenkins M, et al. Failure mode and effects analysis in a paperless radiotherapy department. J Med Imaging Radiat Oncol 2018;62:707-15. [Crossref] [PubMed]
19. Ford EC, Gaudette R, Myers L, et al. Evaluation of safety in a radiation oncology setting using failure mode and effects analysis. Int J Radiat Oncol Biol Phys 2009;74:852-8. [Crossref] [PubMed]
20. Kalapurakal JA, Zafirovski A, Smith J, et al. A comprehensive quality assurance program for personnel and procedures in radiation oncology: value of voluntary error reporting and checklists. Int J Radiat Oncol Biol Phys 2013;86:241-8. [Crossref] [PubMed]