What Are Objective Measures and How Do I Start to Pick Them?
In our first blog of this series, we discussed the model framework for developing a Quality Assessment Process Improvement program. Today, we will discuss how to develop objective measures that are used in three of the aspects of the program, namely Feedback, Data Systems and Monitoring; Systematic Analysis and Systemic Action; and Performance Improvements. The directive to emplo y objective measures is provided in the Centers for Medicare and Medicaid Services (CMS) Conditions of Participation (§482.96(a) Standard: Components of a QAPI Program) that states: “The transplant center’s QAPI program must use objective measures to evaluate the center’s performance with regard to transplantation activities and outcomes”. These objective measures must be in each phase of transplantation (pre-transplant, transplant and post-transplant) as well as donation (pre-donation, donation and post-donation).
What are objective measures? Objective measures are defined as specific data elements that are selected to reflect designated program activities and outcomes. Being objective refers to the fact that the data may be reviewed in an unbiased manner, clearly identified by a finite numerator (how many times the event occurred) and denominator (how many times could it have occurred). Where will the data come from, for example the medical record, the transplant database, or a consent form are questions that should be answered. The measures chosen should be clearly defined for and understood by the program staff so that they know what is being evaluated and why those measures were chosen. The indicators chosen as measures must relate to the core transplant processes that exist across all phases of transplant and living donation as defined by CMS and other regulatory agencies as well as the program’s policies and procedures. They need to relate to the intended and unintended effects resulting from the care provided so that the program can determine whether or not they are meeting their expected goals. After being defined, the objective measures must be collected and analyzed and form the basis of specific recommendations that are communicated to the transplant program decision-makers and hospital leadership. As part of this process, identify who is responsible for collecting or abstracting the data, how frequently the data will be reviewed and what will the triggers be for action, for example, whether it is to create a process improvement project or to retire the measure from active monitoring. If one retires a specific measure, consider monitoring it less frequently, spot checking that particular metric or changing the sampling size to a smaller cohort. This can prevent a previously managed problem from “creeping” up again. Be sure to incorporate these decisions into your transplant QAPI program so that reviewers will be able to assess the design of your QAPI plan.
The structure or context of providing care includes the staff, equipment, facility, finance and organizational culture or environment. All of these factors, recognized or not, contribute to the healthcare process or the procedures, methods, means and series of sequential steps that culminate in specific clinical outcomes, for example patient and graft survival. CMS requires transplant programs to have process measures that reflect the steps required to complete specific tasks as well as outcome measures that relate to the results of the process, or the end point of care.
In healthcare, outcome refers to the results of care (end-point), whether it be positive as in full recovery or negative as in death or infection. Certainly, we are familiar with the transplant outcomes of patient and graft survival and rejection rates that reflect patient well-being and rely on established benchmarks and guidelines such as the SRTR or Scientific Registry of Transplant Recipients for comparison purposes. Once collected, these measures, also representing each phase of transplant and donation, are analyzed and the results disseminated to the program staff and hospital leadership.
One may ask “How do I chose measures to evaluate or will someone tell me what to look at?” The second part is easy, no one will tell you what specific measures to use. As someone said: “If you’ve seen one transplant program, you’ve seen one transplant program…”. As described by Doran in Management Reviewin 1981, use a SMART approach to choose your measures:
If one chooses your measures in this way, you will not be data rich and information poor and will have measures that can result in actionable efforts by your program. To provide some indication of the importance of these items, CMS has provided the following data from surveys conducted in 2014. During this time period, more than 50% of surveyed programs were cited under tag X100 for the following:
No process measures for pre donation and no outcome measures for pre donation, and post donation phases
Measures were not consistently identified by process, outcomes, or by phases
No rationale used for identification of the measure (program unable to state why)
No evidence of data analysis, aggregated data, and appropriate action taken or implemented presented
The QAPI Plan did not include Living Donation in selection of objective measures
Did not identify measures in all programs in all phases as stated in plan
In our next blog, we will talk more specifically about how to manage these objective measures and establish process improvement projects so as to prevent citations such as those above.
About the Author: Dr. John Daller, MD, PhD, FACS
As a former Director of several transplant programs, Dr. Daller has expertise in all aspects of transplant program management, as well as hospital program development including clinical, regulatory, business and administrative leadership via his company Strategic Illuminations. He consults in the area of medical legal review, due diligence and scientific evaluations, as well as utilization review via Daller Consulting. He is also Chief Medical Officer of Concordia Valsource, LLC which provides consultative services to developing biopharmaceutical companies and to Venture Capital groups investing in the health and life sciences. Previously, Dr. Daller was Vice President for Medical Programs in the Transplant Business Unit of Genzyme Corporation.