Evaluation and tuning of a commercial automated planning system for prostate radiotherapy.

Background and Purpose: Artificial Intelligence (AI) based automated planning is increasing in popularity. Guidance has been published recommending approaches for the safe implementation and monitoring of these techniques. An evaluation of a commercial AI tool was undertaken and reported along with the specific methods used to evaluate, tune and monitor the plan quality. Materials and Methods: A total of 335 previously planned prostate patients were used to evaluate and commission a commercial AI based planning solution. The quality of the automatically produced plans was compared to previous practice using models inspired by existing research into knowledge based planning. A quantile regression based technique identified the most optimal historic plans to be used for model tuning. Finally, a control chart based method was validated for post-deployment monitoring of the produced plan quality. Results: The baseline model provided by the manufacturer was found to provide good plan quality overall with 9 out 15 plan quality metrics found to have lower variance after accounting for anatomy. Rectum sparing was found to be inferior to human generated plans. Two further iterations of the model were produced in collaboration with the manufacturer. Further iterations of the model resulted in comparable rectum sparing, a 0.02 mean difference in rectum V50 Gy, was achieved whilst maintaining much of the improved consistency. Conclusions: A method to implement the guidance for commissioning of automated and AI based planning tools is presented alongside a method for monitoring the subsequent plan quality. The final plan quality achieved was comparable to or better than the original plans following two revisions.