Precision of medication therapy problem identification and classification amongst primary care clinic pharmacists

View Image


Comprehensive medication management (CMM) offers a strategic and stepwise process for identifying medication therapy problems (MTPs). Precision and consistency in practice among pharmacists providing CMM services is critical to the success of an effective and quality CMM program, especially if system-wide beneficial outcomes are to be observed. The purpose of this study is to assess the level of agreement on MTP identification and classification between primary care, ambulatory care pharmacists within a health-system that has recently implemented system-wide pharmacist provision of CMM services.


Twenty standardized case vignettes were created and distributed to pharmacists who were asked to review the cases and identify and categorize MTPs. All primary care, ambulatory care pharmacists from the health-system were included. Case vignettes were created using information from patient electronic medical records with a goal to emulate realistic patient scenarios and contain the information necessary for pharmacists to perform a CMM review, as is done in practice. Private health information was omitted from all cases. Prior to distribution, each case was reviewed for content and clinical significance by an independent pharmacist with expertise in ambulatory care practice and CMM. Pharmacist respondents were all given the same 20 case vignettes and were instructed to categorize MTPs according to the nationally and health-system adopted Pharmacy Quality Alliance’s (PQA) framework. Primary outcomes include number of MTPs identified, identification (yes/no) of specific MTP types (e.g., need for a statin), and PQA category used when classifying MTPs. The precision or level of agreement on MTP identification/categorization was measured using intraclass correlation coefficient (ICC), computed with a two-way random effects model, using the individual, absolute agreement form of the ICC. ICC point estimates were interpreted using the Landis and Koch guideline. This study was deemed exempt by an institutional review board.


The study was conducted in 19 pharmacists from 12 different primary care clinics. In regard to the number of MTPs identified by pharmacists, “moderate agreement” was observed (ICC equal to 0.45; 95% confidence interval [CI]: 0.31 to 0.65). In approximately one-half of opportunities, the pharmacists agreed perfectly on the number of MTPs; in approximately one-third of opportunities, the number of MTPs identified varied by 1; and approximately one-tenth of the time, the number of MTPs varied by 2. At most, the number varied by 5, but this occurred in less than 1% of all opportunities. In regard to the identification (yes/no) of specific MTP types and PQA category used to classify, the results varied by the specific MTP types. For example, in all case vignettes in which pharmacists identified that a patient should be initiated on a statin medication, “fair agreement” was observed in identifying that MTP (ICC equal to 0.24; 95% CI: 0.09 to 0.60) with pharmacists agreeing in 74% of opportunities. In regard to the level of agreement when classifying the statin MTP, of those that identified that MTP, “perfect agreement” was observed in the specific PQA category used.


This study provides valuable information on the level of agreement on MTP identification and classification between primary care, ambulatory care pharmacists. Pharmacists demonstrated moderate agreement on the number of MTPs identified per case vignette with variable levels of agreement on identification/classification of specific MTP types. These results support the need for further training and education and provide the information necessary to target specific disease states and MTP types; this will allow pharmacists to provide more consistent patient care and maintain high fidelity to the CMM practice model.

Recommend0 recommendationsPublished in College of Pharmacy, Virtual Poster Session Spring 2021


  1. Good looking poster! Could you elaborate as to why the ICC and agreement percentages don’t seem to match up? I’m mostly curious to get a better sense of how that type of analysis works. Thanks!

    1. Thank you! While, I don’t pretend to be an expert on these statistical methods (recognizing the help from Greg Stoddard, our statistician), I will do my best to answer your question.

      The ICC and % agreement don’t always appear to match up because of how the ICC is calculated. ICC is a measure of how well the pharmacists agree beyond chance. When there is low variation in the agreement, or when there is a low sample size, the statistic does not calculate well. This is particularly prone to happen when the value is binary (as in our case). In these cases, as recommended by our statistician, it is more useful to use the percent agreement.

  2. Great project and poster Bryce! I find it really interesting that smoking cessation had the highest agreement level! I’m curious about next steps- if you were to design a CMM education session for the participating pharmacists what specifically would you target to improve these scores in the future?

    1. I think that is a great question. I think apart from a CE-styled presentation on targeting disease states, it would be interesting to have pharmacists in pairs or groups (splitting up practice partners) work through targeted cases together and gain each-others insights. I think just encouraging that clinical discussion amongst the whole team would lead to more consistency. Another possibility would be cross-training/coverage days at other clinics with other pharmacists. I think this would certainly be more logistically challenging, but again may offer the insight to how other members of the team identify and categorize problems.

  3. The breakdown of the medical issues in Figure 1 bring needed solutions to light. Thank you.

Comments are closed.