Advertisement

Automated Prediction of Risk for Problem Opioid Use in a Primary Care Setting

Published:January 29, 2015DOI:https://doi.org/10.1016/j.jpain.2015.01.011

      Highlights

      • An automated method predictive in identifying clinician-documented problem opioid use was assessed.
      • Analysis developed measures using information from a health plan's electronic health record system.
      • Models showed moderate sensitivity and specificity in predicting problem opioid use.
      • The method may enable more efficient implementation of problem opioid use detection.

      Abstract

      Identification of patients at increased risk for problem opioid use is recommended by chronic opioid therapy (COT) guidelines, but clinical assessment of risks often does not occur on a timely basis. This research assessed whether structured electronic health record (EHR) data could accurately predict subsequent problem opioid use. This research was conducted among 2,752 chronic noncancer pain patients initiating COT (≥70 days' supply of an opioid in a calendar quarter) during 2008 to 2010. Patients were followed through the end of 2012 or until disenrollment from the health plan, whichever was earlier. Baseline risk indicators were derived from structured EHR data for a 2-year period prior to COT initiation. Problem opioid use after COT initiation was assessed by reviewing clinician-documented problem opioid use in EHR clinical notes identified using natural language processing techniques followed by computer-assisted manual review of natural language processing–positive clinical notes. Multivariate analyses in learning and validation samples assessed prediction of subsequent problem opioid use. The area under the receiver operating characteristic curve (c-statistic) for problem opioid use was .739 (95% confidence interval = .688, .790) in the validation sample. A measure of problem opioid use derived from a simple weighted count of risk indicators was found to be comparably predictive of the natural language processing measure of problem opioid use, with 60% sensitivity and 72% specificity for a weighted count of ≥4 risk indicators.

      Perspective

      An automated surveillance method utilizing baseline risk indicators from structured EHR data was moderately accurate in identifying COT patients who had subsequent problem opioid use.

      Key words

      Problem use of prescription opioids exacts an increasing clinical, economic, and societal burden within the United States.
      • Birnbaum H.G.
      • White A.G.
      • Schiller M.
      • Waldman T.
      • Cleveland J.M.
      • Roland C.L.
      Societal costs of prescription opioid abuse, dependence, and misuse in the United States.
      Institute of Medicine of the National Academies
      Relieving Pain in America: A Blueprint for Transforming Prevention, Care, Education, and Research.
      • White A.G.
      • Birnbaum H.G.
      • Mareva M.N.
      • Daher M.
      • Vallow S.
      • Schein J.
      • Katz N.
      Direct costs of opioid abuse in an insured population in the United States.
      Recent calls to action challenge clinicians and health care organizations to ensure patient safety while offering effective care for chronic pain.
      Office of National Drug Control Policy
      Epidemic: Responding to America’s Prescription Drug Abuse Crisis.
      Although clinical guidelines recommend a “Universal Precautions” approach to opioid prescribing that includes timely assessment of risks for problem opioid use, available evidence indicates that timely risk assessment usually does not occur.
      • Chou R.
      • Fanciullo G.J.
      • Fine P.G.
      • Adler J.A.
      • Ballantyne J.C.
      • Davies P.
      • Donovan M.I.
      • Fishbain D.A.
      • Foley K.M.
      • Fudin J.
      • Gilson A.M.
      • Kelter A.
      • Mauskop A.
      • O'Connor P.G.
      • Passik S.D.
      • Pasternak G.W.
      • Portenoy R.K.
      • Rich B.A.
      • Roberts R.G.
      • Todd K.H.
      • Miaskowski C.
      American Pain Society-American Academy of Pain Medicine Opioids Guidelines Panel. Opioid Treatment Guidelines
      Clinical guidelines for the use of chronic opioid therapy in chronic noncancer pain.
      • Gourlay D.L.
      • Heit H.A.
      • Almahrezi A.
      Universal precautions in pain medicine: A rational approach to the treatment of chronic pain.
      Washington State Agency Medical Directors’ Group
      Interagency Guideline on Opioid Dosing for Chronic Non-cancer Pain: An Educational Aid to Improve Care and Safety With Opioid Treatment.
      • Krebs E.E.
      • Ramsey D.
      • Miloshoff J.M.
      • Bair M.J.
      Primary care monitoring of long-term opioid therapy among veterans with chronic pain.

      Prescription Drug Monitoring Program (PDMP) Center of Excellence, Brandeis University. National Rx Drug Abuse Summit Meeting, 22 April 2014

      • Rolfs R.T.
      • Johnson E.
      • Williams N.J.
      • Sundwall D.N.
      Utah clinical guidelines on prescribing opioids for treatment of pain.
      • Sehgal N.
      • Manchikanti L.
      • Smith H.S.
      Prescription opioid abuse in chronic pain: A review of opioid abuse predictors and strategies to curb opioid abuse.
      • Starrels J.L.
      • Becker W.C.
      • Alford D.P.
      • Kapoor A.
      • Williams A.R.
      • Turner B.
      Systematic review: Treatment agreements and urine drug testing to reduce opioid misuse in patients with chronic pain.
      • Von Korff M.
      • Kolodny A.
      • Deyo R.A.
      • Chou R.
      Long-term opioid therapy reconsidered.
      Rigorous longitudinal research evaluating the prediction of subsequent problem opioid use is also lacking.
      • Chou R.
      • Fanciullo G.J.
      • Fine P.G.
      • Miaskowski C.
      • Passik S.D.
      • Portenoy R.K.
      Opioids for chronic noncancer pain: Prediction and identification of aberrant drug-related behaviors: A review of the evidence for an American Pain Society and American Academy of Pain Medicine clinical practice guideline.
      Accurate identification of patients most likely at risk for problem opioid use is a prerequisite for widely recommended approaches to mitigating these risks. Multiple patient- and clinician-based screeners exist to assess potential problem opioid use risks.
      • Chou R.
      • Fanciullo G.J.
      • Fine P.G.
      • Miaskowski C.
      • Passik S.D.
      • Portenoy R.K.
      Opioids for chronic noncancer pain: Prediction and identification of aberrant drug-related behaviors: A review of the evidence for an American Pain Society and American Academy of Pain Medicine clinical practice guideline.
      • Jones T.
      • Moore T.
      • Levy J.L.
      • Daffron S.
      • Browder J.H.
      • Allen L.
      • Passik S.D.
      A comparison of various risk screening methods in predicting discharge from opioid treatment.
      • Moore T.M.
      • Jones T.
      • Browder J.H.
      • Daffron S.
      • Passik S.D.
      A comparison of common screening methods for predicting aberrant drug-related behavior among patients receiving opioids for chronic pain management.
      • Passik S.D.
      • Kirsh K.L.
      • Casper D.
      Addiction-related assessment tools and pain management: Instruments for screening, treatment planning, and monitoring compliance.
      • Turk D.C.
      • Swanson K.S.
      • Gatchel R.J.
      Predicting opioid misuse by chronic pain patients: A systematic review and literature synthesis.
      However, there are deficiencies in evaluation of the accuracy of these screeners in predicting risks for subsequent onset of problem opioid use, so there is no consensus about the value of screening or which screeners are most useful.
      • Moore T.M.
      • Jones T.
      • Browder J.H.
      • Daffron S.
      • Passik S.D.
      A comparison of common screening methods for predicting aberrant drug-related behavior among patients receiving opioids for chronic pain management.
      • Passik S.D.
      • Kirsh K.L.
      • Casper D.
      Addiction-related assessment tools and pain management: Instruments for screening, treatment planning, and monitoring compliance.
      • Turk D.C.
      • Swanson K.S.
      • Gatchel R.J.
      Predicting opioid misuse by chronic pain patients: A systematic review and literature synthesis.
      These screeners assess patient risk factors including substance use disorder history, family history of substance use disorder, and significant psychological problems. However, it has proven difficult to implement routine risk assessment in community practice for several reasons. Busy clinicians working in time-constrained care settings may not administer screeners because of competing demands and clinical priorities among often complex chronic pain patients. And it is often unclear when patients transition from short-term to long-term use of opioids, so formal evaluation visits to assess the appropriateness of chronic opioid therapy (COT) do not reliably occur.
      • Krebs E.E.
      • Ramsey D.
      • Miloshoff J.M.
      • Bair M.J.
      Primary care monitoring of long-term opioid therapy among veterans with chronic pain.
      Automated assessment utilizing structured electronic health records (EHRs) data could address several of these difficulties, thereby facilitating more timely and consistent opioid risk assessment by clinicians and health care systems. Recently, predictive models of opioid abuse have used insurance claims data to assess the likelihood of problem opioid use, with initially promising results.
      • Dufour R.
      • Joshi A.V.
      • Pasquale M.K.
      • Schaaf D.
      • Mardekian J.
      • Andrews G.A.
      • Patel N.
      The prevalence of diagnosed opioid abuse in commercial and Medicare managed care populations.
      • Rice J.B.
      • White A.G.
      • Birnbaum H.G.
      • Schiller M.
      • Brown D.A.
      • Roland C.L.
      A model to identify patients at risk for prescription opioid abuse, dependence, and misuse.
      • Sullivan M.D.
      • Edlund M.J.
      • Fan M.Y.
      • Devries A.
      • Brennan Braden J.
      • Martin B.C.
      Risks for possible and probable opioid misuse among recipients of chronic opioid therapy in commercial and medicaid insurance plans: The TROUP Study.
      However, these studies relied solely upon recorded International Classification of Diseases, Ninth Revision (ICD-9), diagnostic codes to ascertain opioid abuse and were of short duration (≤6 months).
      The increasing availability of data from patients' EHRs including clinicians' notes offers new opportunities to develop predictive models for assessing potential risks for problem opioid use. In this research, we generated and evaluated a measure of problem opioid use based on natural language processing (NLP) techniques utilizing information contained within clinicians' notes in the EHR pertaining to COT patients.
      The aim of this article is to report on a predictive model developed to assess the likelihood of problem opioid use over a 2- to 5-year period following initiation of COT within a large health plan. We assessed whether readily accessible baseline risk indicators obtained from structured EHR data at initiation of long-term opioid therapy were able to accurately predict subsequent problem opioid use.

      Methods

       Setting and Data

      This study was conducted at Group Health Cooperative (GHC), a large, mixed-model health plan established in 1945 that now serves approximately 600,000 patients in urban, suburban, and rural areas of Washington State. Group Health is composed of both integrated group practice (IGP; salaried physicians working in GHC clinics) and network (fee-for-service physicians working in community settings) segments. This research is limited to members in the IGP because the study relies on clinical electronic medical records data that are available only in the IGP. The GHC uses the Epic EHR system (http://www.epic.com/; Epic, Verona, WI) to document care delivery, including primary, specialist, and emergency care encounters and hospital discharge summaries. The GHC Epic EHR system is the digital chart used for patient care within the IGP, having replaced the paper chart in 2005. The GHC maintains structured data, which are data elements entered into the GHC Epic system by health care staff using predefined options (eg, diagnostic codes), including information on most aspects of care delivery such as patient demographics, diagnostic codes, procedures, medications, and laboratory results. In addition, the GHC Epic EHR system includes free text clinical notes entered by health care staff during patient encounters and other contacts with patients. Approval for this study was granted by the Group Health Human Subjects institutional review board.
      The sample consisted of GHC members aged 18 years or older who initiated COT for noncancer pain between 2008 and 2010. COT was defined as receipt of ≥70 days' supply of transdermal or oral opioids (except buprenorphine) in a calendar quarter, which corresponded to >75% of the days in the quarter covered by an opioid prescription. We employed this definition because it corresponded to the operational definition of COT employed by GHC for a COT risk reduction initiative implemented in 2010. Because analyses examined opioid use over multiple years, we counted the number of quarters that study patients met this criterion for receiving COT. To ensure sustained use of opioids and a sufficient COT duration to permit clinician identification of problem opioid use, we required that study patients receive at least 2 quarters of COT within a 1-year period. We identified the first calendar quarter between 2008 and 2010 in which a subject received COT (index quarter). If the subject received at least 1 more quarter of COT in the 3 quarters following the index quarter, she or he was provisionally eligible for the study. To restrict the sample to those initiating COT between 2008 and 2010, we excluded subjects who received COT in any quarter in 2006 and 2007. Further, subjects were required to be enrolled in the health plan at least 6 of the 8 calendar quarters before the index quarter and 6 of the 8 quarters following the index quarter (inclusive) so each subject had a 2-year “pre-period” and at least a 2-year follow-up period. These enrollment requirements ensured the availability of EHR data to capture utilization prior to the initiation of COT use and the potential availability of clinical notes data to document problem opioid use after COT initiation. To restrict the study to patients receiving COT for noncancer pain, patients were excluded if they had ≥2 visits with cancer diagnoses (excluding nonmelanoma skin cancer) during any calendar 1-year period between 2006 and 2012 or had received an opioid prescription from an oncologist or were admitted to hospice during the study period. Based on a random assignment performed on the entire pool of subjects potentially eligible for the study (eg, before application of study inclusion/exclusion criteria), patients were grouped into “learning” and “validation” samples.

       NLP Measure of Problem Opioid Use

      To construct a measure of clinician-documented problem opioid use, we began with a definition of opioid misuse or abuse using the National Institute of Drug Abuse definition: “Prescription drug abuse is the use of a medication without a prescription, in a way other than as prescribed, or for the experience or feelings elicited.” We defined clinician-documented problem opioid use as a practitioner statement in the clinical record of a diagnosis or assessment that the subject is overusing, is misusing, is abusing, or has become addicted to prescription opioids. Relevant documentation was in the form of an explicit description in the narrative of a clinical note or a coded entry into the subject's active problem list. In this prospective study, problem opioid use had to be documented during the individual's follow-up period. A person's follow-up period extended from the index quarter through December 31, 2012, or his or her disenrollment from the health plan, whichever was earlier.
      Because of the large number of clinical notes available for each subject (on average, 355 clinical notes per patient), we used NLP techniques to identify subjects who had text in the clinician's notes that was potentially indicative of problem opioid use. NLP uses computing techniques to automate identification of meaningful content in unstructured text, facilitating discovery of such content within large numbers of documents. NLP methods can address linguistic variation (different words with the same meaning), polysemy (single words with several meanings), negation (eg, “reports no pain” vs “reports severe pain”), ambiguity, and temporality. NLP methods can be particularly useful in searching the free text clinical notes of the EHR.
      We performed 3 phases of NLP searching through all the relevant clinical notes for each subject (2008 through 2012). First, the algorithm identified subjects who had clinical notes that indicated opioid overuse, misuse, abuse, or addiction. Second, the algorithm identified subjects who had clinical notes that indicated opioid “dependence,” which sometimes indicated only physiologic dependence on opioids. Third, we identified any additional subjects who had an ICD-9 code for opioid abuse or dependence [304.0, .00, .01, .02; 304.7, .70, .71, .72; 305.5, .50, .51, .52] in structured EHR data who were not identified in the prior 2 phases as having clinician-documented opioid overuse, misuse, or abuse. Complete details of the development of computer-assisted NLP techniques applied in this study are described elsewhere.
      • Cronkite D.
      Estimating Prevalence of Clinician-Labeled Opioid Abuse: Structured vs. Unstructured Clinical Data. HMO Research Network Annual Meeting Presentation.
      Two pairs (M.V.K. and a research specialist, and M.V.K. and K.S.) of Group Health Research Institute chart reviewers then used a customized computer program that highlighted relevant text in NLP-identified clinical notes to conduct a manual validation of all subjects in the study sample for whom the NLP algorithm indicated positive for problem opioid use (NLP-identified positive), reviewing up to 4 NLP-identified clinical notes per subject. Before the manual validation was begun, interrater reliability between the pairs of reviewers was tested and was found to be high (kappa = .86, 97% agreement for the first pair of raters [M.V.K. and a research specialist] on a random sample of 50 NLP-identified positive subjects; kappa = .71, 88% agreement for the second pair of raters [M.V.K. and K.S.] on a random sample of 137 NLP-identified positive subjects).

       Baseline Risk Indicators

      A set of baseline risk indicators was constructed from the structured data in the EHR to include in the multivariate analyses used to evaluate each measure of problem opioid use. These baseline risk indicators were taken from the GHC Epic EHR system and developed from structured data (not the clinical notes) entered by the GHC health care staff during the patient's visits occurring in the 2-year “pre-period” and are likely to be correlated with problem opioid use based on earlier literature.
      • Dufour R.
      • Joshi A.V.
      • Pasquale M.K.
      • Schaaf D.
      • Mardekian J.
      • Andrews G.A.
      • Patel N.
      The prevalence of diagnosed opioid abuse in commercial and Medicare managed care populations.
      Baseline risk indicators included in the model were as follows: age (18–44, 45–64, and ≥65 [as of the index quarter]), gender (female/male), ethnicity (white, nonwhite, unknown), smoking status (equal to 1 as a current smoker if indicated at any time during the study period, 0 otherwise), ICD-9 diagnosis of opioid abuse/dependence (equal to 1 if indicated in any quarter during the 2-year “pre-period” and 0 otherwise; this variable was obtained from the structured EHR data available on the patient and is different from the ICD-9 code “mention” in the clinician's notes used for the NLP-derived measure of problem opioid use), ICD-9 diagnosis of nonopioid prescription drug abuse/dependence (equal to 1 if indicated in any quarter during the 2-year “pre-period” and 0 otherwise), ICD-9 diagnosis of a mental disorder (equal to 1 if indicated in any quarter during the 2-year “pre-period” and 0 otherwise; mental disorders included depression, bipolar disorder, anxiety, autism, schizophrenia, dementia, other psychosis, and self-inflicted injury [including “possible”]), ICD-9 diagnosis of alcohol abuse/dependence (equal to 1 if indicated in any quarter during the 2-year “pre-period” and 0 otherwise), and ICD-9 diagnosis of hepatitis C (equal to 1 if indicated in any quarter during the 2-year “pre-period” and 0 otherwise).

       Weighted Count of Risk Indicators

      We also developed a simple, weighted count of risk indicators to determine if it predicted NLP-ascertained problem opioid use. We did this because the use of a logistic regression model to predict the likelihood of future problem opioid use would be difficult in many community practice settings. The weighted count of risk indicators could be readily implemented in any setting able to determine the status of patients on each of the risk indicators from medical records data.
      To construct the weighted count of risk indicators, we preidentified relevant risk indicators, including age, smoking status, and ICD-9 codes in the structured data (not the clinical notes) that appeared in the 2-year period prior to COT initiation. These diagnoses included opioid abuse/dependence, nonopioid drug abuse/dependence, alcohol abuse/dependence, mental disorders, and hepatitis C. For each risk indicator, we estimated a separate logistic regression model, with each predictor entered individually as the independent variable and problem opioid use as the dependent variable. We assigned weights to each of the risk indicators based on the odds ratio (OR) from the logistic regression models according to the following criteria: for 1 < OR ≤ 2.5, weight = 1; for 2.5 < OR ≤ 5, weight = 2; and for OR > 5, weight = 3. The following weights were employed: for age 18 to 44, weight = 3; for age 45 to 64, weight = 1; for prior opioid abuse/dependence diagnosis, weight = 3; for prior nonopioid drug abuse/dependence, weight = 3; for prior alcohol abuse/dependence diagnosis, weight = 1; for prior mental disorder diagnosis, weight = 2; for current smoking, weight = 2; and for prior hepatitis C diagnosis, weight = 2. We then summed these weights for each subject. The weighted count of risk indicators was tested in the learning, validation, and combined samples. This weighted count was also used to assess whether a prediction model estimated in the learning sample achieved comparable prediction in the validation sample.

       Multivariate Analyses

      We performed logistic regression analysis on the learning, validation, and combined samples to evaluate the ability to predict subsequent problem opioid use using the NLP classification of problem opioid use. We also assessed the ability of the weighted count of risk indicators to predict the NLP classification of problem opioid use utilizing logistic regression. Receiver operating characteristic (ROC) curves were generated, and corresponding c-statistics and 95% confidence intervals (CIs) were calculated for each logistic regression. The c-statistic measures the discriminatory ability of the model, with a greater area under the curve representing better prediction, where .50 indicates prediction no better than chance and 1.00 indicates perfect prediction. We also estimated the weighted count measure model's sensitivity and specificity and the positive predictive value (PPV, defined as the number of patients predicted positive for problem opioid use by the weighted count measure who were also positive on the NLP classification of problem opioid use, divided by the total number of patients) and negative predictive value (NPV, defined as the number of patients predicted to not have problem opioid use by both the weighted count measure and NLP classification, divided by the total number of patients).

       Software

      We developed a regular expression-based NLP system in the Python programming language, version 2.7 (Python Software Foundation; http:www.python.org), modeled after the named entity recognition in the widely used Apache Clinical Text Analysis and Knowledge Extraction System (cTAKES; The Apache Software Foundation; http:www.apache.org).
      • Savova G.K.
      • Masanz J.J.
      • Ogren P.V.
      • Zheng J.
      • Sohn S.
      • Kipper-Schuler K.C.
      • Chute C.G.
      Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): Architecture, component evaluation and applications.
      The system uses a list of phrases to identify mentions of problem opioid use in clinical notes, accounting for negated, uncertain, hypothetical, and historical language qualifiers, references to persons other than the patient, and spelling errors. Additionally, the Python GUI library wxPython 2.8 was used to facilitate the manual validation of clinician-documented problem opioid use with highlighting of key terms. All analytic file construction, descriptive statistics, and multivariate analyses (including statistical calculations) were performed using SAS, version 9.3 (SAS Institute, Cary, NC).

      Results

      Overall, the study had 24,716 potential subjects. Of these, 10.4% (n = 2,574) had 1 of the cancer exclusions, and 43% (n = 9,475) of the remaining 22,142 subjects were excluded from this analysis because they had received at least 1 quarter of COT in 2006 or 2007. Of the remaining 12,667 subjects, 57.9% (n = 7,339) were excluded because they did not meet our definition of COT between 2008 and 2010, and an additional 2,576 were excluded because of failure to satisfy the pre- and/or post-index enrollment criteria, resulting in a total of 2,752 COT subjects who were eligible for this study. The original pool of 24,716 subjects had been assigned at random to a learning sample or to a validation sample. Among the 2,752 subjects eligible for the current analysis, 1,449 had been randomly assigned to the learning sample and 1,303 to the validation sample.
      Approximately half of the 2,752 study subjects were 45 to 64 years of age (see Table 1). Close to two-thirds were women, whereas one-third were smokers. Of note, less than 1% had a structured ICD-9 diagnosis of opioid abuse/dependence recorded in structured EHR data in the 2 years prior to COT initiation. Among the 2,752 COT initiators, 158 (5.7%) were identified as positive for subsequent problem opioid use over the follow-up period based on the text contained within the clinicians' notes portion of the EHR, using the NLP classification and computer-assisted manual validation. The median weighted count of risk indicators was equal to 3.
      Table 1Sample Characteristics (N = 2,752)
      Variable% or Mean (SD)n
      Age
      Assessed at start of study period.
       18–4421.2583
       45–6449.91,373
       ≥6528.9796
      Gender
       Male37.71,039
       Female62.31,713
      Ethnicity
       Non-Hispanic white77.82,141
       Nonwhite13.3365
       Unknown8.9246
      Current smoker
       Yes33.3916
       No33.71,836
      ICD-9 diagnosis of opioid abuse/dependence
      Assessed in the 2-year pre-period prior to the index quarter.
       Yes.925
       No99.12,727
      ICD-9 diagnosis of drug abuse/dependence (excluding opioids)
      Assessed in the 2-year pre-period prior to the index quarter.
       Yes2.158
       No97.92,694
      ICD-9 diagnosis of alcohol abuse/dependence
      Assessed in the 2-year pre-period prior to the index quarter.
       Yes4.9135
       No95.12,617
      ICD-9 diagnosis of mental health disorder
      Assessed in the 2-year pre-period prior to the index quarter.
       Yes43.21,189
       No56.81,563
      ICD-9 diagnosis of hepatitis C
      Assessed in the 2-year pre-period prior to the index quarter.
       Yes4.8132
       No95.22,620
      Mean no. of quarters received ≥30 days' supply
      Assessed between January 1, 2008, and the end of follow-up period.
      12.2 (5.0)
      Mean no. of quarters received ≥70 days' supply
      Assessed between January 1, 2008, and the end of follow-up period.
      8.8 (4.8)
      Mean (median) number of quarters in follow-up period13.4 (13)
      Proportion (number) clinician-documented problem opioid use
      Assessed between the index quarter and the end of follow-up period.
       Yes5.7158
       No94.32,594
      Weighted count of risk indicators
       015.1415
       118.1497
       210.9299
       325.8709
       43.289
       517.3476
       61.540
       75.9161
       ≥82.466
      Mean (median) weighted count of risk indicators2.90 (3.00)
      Range of weighted count of risk indicators0–16
      Assessed at start of study period.
      Assessed in the 2-year pre-period prior to the index quarter.
      Assessed between January 1, 2008, and the end of follow-up period.
      § Assessed between the index quarter and the end of follow-up period.
      Table 2 reports the multivariate model results for the NLP measure of problem opioid use in the learning (n = 1,449), validation (n = 1,303), and combined (N = 2,752) samples. In the combined sample, variables indicating age 18 to 44, opioid abuse diagnosis, positive smoking status, and mental disorder diagnoses during the 2-year prior period were significant predictors. It should be noted that probably because of small numbers with the risk indicator, the results for opioid abuse diagnosis were not significant in the learning sample. ROC analysis yielded an area under the curve (c-statistic) of .754 (CI = .689, .819), .739 (CI = .688, .790), and .745 (CI = .705, .785) for the learning, validation, and combined samples, respectively.
      Table 2Multivariate Logistic Regression of NLP Measure of Problem Opioid Use
      Baseline VariableLearning Sample (n = 1,449)Validation Sample (n = 1,303)Combined Sample (N = 2,752)
      Age
       18–444.58 (1.99, 10.53)4.41 (2.22, 8.74)4.54 (2.68, 7.69)
       45–641.20 (.52, 2.80)1.42 (.72, 2.82)1.33 (.78, 2.26)
       ≥65 (referent)1.01.01.0
      Smoking status
       Current1.41 (.81, 2.46)1.57 (.99, 2.49)1.47 (1.03, 2.08)
       Past/never (referent)1.01.0
      ICD-9 diagnosis of opioid abuse/dependence
       Yes2.90 (.59, 14.16)16.33 (4.12, 64.73)7.44 (2.79, 19.80)
       No (referent)1.01.01.0
      ICD-9 diagnosis of drug abuse/dependence (excluding opioids)
       Yes2.12 (.69, 6.55)1.75 (.55, 5.58)1.63 (.73, 3.65)
       No (referent)1.01.01.0
      ICD-9 diagnosis of alcohol abuse/dependence
       Yes.87 (.30, 2.53)1.32 (.53, 3.25)1.02 (.51, 2.03)
       No (referent)1.01.01.0
      ICD-9 diagnosis of mental health disorder
       Yes1.84 (1.05, 3.21)1.61 (1.02, 2.53)1.73 (1.22, 2.45)
       No (referent)1.01.01.0
      ICD-9 diagnosis of hepatitis C
       Yes2.55 (1.00, 6.45)1.10 (.42, 2.92)1.56 (.80, 3.05)
       No (referent)1.01.01.0
      NOTE. Values are OR (95% CI).
      We then assessed the ability of the weighted count of risk indicators to predict the NLP classification of problem opioid use. Fig 1 reports the ROC curves of this model in both the learning and validation samples. The ROC area under the curve (c-statistic) was .733 (CI = .666, .800), .718 (CI = .668, .768), and .724 (CI = .684, .764) for the learning, validation, and combined samples, respectively.
      Figure thumbnail gr1
      Figure 1ROC curves for learning (n = 1,449) and validation (n = 1,303) samples for predicting problem opioid use using the weighted count measure.
      We also calculated the overall sensitivity and specificity of the weighted count of risk indicators model by applying the estimates from the logistic regression model that predicted problem opioid use in the learning sample to the validation sample. The sensitivity of the regression model prediction of problem opioid use was 58.3%, and specificity was 71.2% at a predicted probability cutoff value equal to .046. This predicted probability corresponded to a count of ≥4 risk indicators.
      We also estimated sensitivity, specificity, PPV, and NPV at varying cut points for the weighted count of risk indicators. These estimates are reported in Table 3 for the combined study sample. A weighted count of risk indicators of ≥3 provided good sensitivity (82.8%) but poor specificity (45.6%). A weighted count of risk indicators of ≥4 provided moderate predictive ability, with fair specificity (71.6%) balanced with lower sensitivity (60.1%). Table 4 provides more detail on how sensitivity, specificity, PPV, and NPV were calculated for a weighted count of risk indicators of ≥4. Using a cutoff of ≥4, the PPV was 11.4% (95/832), and the NPV was 96.7% (1,857/1,920).
      Table 3Prediction of Clinician-Documented Problem Opioid Use at Various Cut Points (N = 2,752)
      Weighted Count of Risk Indicators CutoffPPVNPVSensitivitySpecificity
      ≥16.698.896.815.8
      ≥27.898.490.534.6
      ≥38.597.882.945.6
      ≥411.496.760.171.6
      ≥619.595.732.991.7
      ≥827.394.811.498.2
      NOTE. Values are percentages.
      Table 4Agreement of Classification of Problem Opioid Use: NLP-Based Classification and Weighted Count Risk Screener
      Weighted Count Risk Screener of Problem Opioid Use
      With classification of problem opioid use based on the weighted count measure with a cut point of ≥4.
      NLP-Based Classification of Problem Opioid Use
      AbsentPresentTotal
      0–3 risk indicators1,857631,920
      ≥4 risk indicators73795832
      Total2,5941582,752
      NOTE. Sensitivity: 95/158 = 60.1%; specificity: 1,857/2,594 = 71.6%; PPV: 95/832 = 11.4%; NPV: 1,857/1,920 = 96.7%.
      With classification of problem opioid use based on the weighted count measure with a cut point of ≥4.

      Discussion

      Our results suggest the potential utility of automated surveillance to assess risks of problem opioid use utilizing EHR data as predictors. Performance of both the NLP measure model and the weighted count of risk indicators was similar to that of widely used risk screeners in other disease states (eg, cardiovascular disease predictive models have reported area under the curve c-statistics in the range of .68–.77).
      • Wilson P.
      • D’Agostino R.
      • Levy D.
      • Belanger A.
      • Silbershatz H.
      • Kannel W.
      Prediction of coronary heart disease using risk factor categories.
      The observed sensitivities and specificities using the weighted count of risk indicators reflect only fair prediction of the risk for subsequent problem opioid use as assessed by the NLP measure.
      If we view the NLP classification as a “gold standard” for problem opioid use, the 60% sensitivity means that about 4 of 10 patients who have subsequent problem opioid use would be predicted as negative for subsequent problem opioid use when applying the NLP classification approach. The 72% specificity means that 28% of patients who are negative for subsequent problem opioid use in clinical notes would be predicted as positive when applying the NLP classification approach. Although the performance of the NLP classification approach is similar to that of tests widely used in clinical practice, the attenuated sensitivities and specificities mean that initial screening cannot accurately predict who will and who will not subsequently develop problem opioid use.
      • Jones T.
      • Moore T.
      • Levy J.L.
      • Daffron S.
      • Browder J.H.
      • Allen L.
      • Passik S.D.
      A comparison of various risk screening methods in predicting discharge from opioid treatment.
      • Moore T.M.
      • Jones T.
      • Browder J.H.
      • Daffron S.
      • Passik S.D.
      A comparison of common screening methods for predicting aberrant drug-related behavior among patients receiving opioids for chronic pain management.
      These results suggest that it may be feasible to develop simple risk stratification algorithms to initially alert clinicians to potential patients at higher risk for problem opioid use, with the caveat that such algorithms may have far from optimal sensitivity and specificity for predicting who will develop subsequent problem opioid use. Because a variety of factors often prevent timely assessment of risk factors for problem opioid use among patients initiating COT in community practice settings, automated screening may help to initially identify some patients at elevated risk for subsequent problem opioid use for whom further evaluation may be warranted.
      • Krebs E.E.
      • Ramsey D.
      • Miloshoff J.M.
      • Bair M.J.
      Primary care monitoring of long-term opioid therapy among veterans with chronic pain.
      Although the risk indicators from EHR data tested in this research are similar to those included in patient-administered risk assessment tools, it is not possible to determine the performance of the automated tool evaluated here relative to the patient-administered scales tested in prior research.
      • Chou R.
      • Fanciullo G.J.
      • Fine P.G.
      • Miaskowski C.
      • Passik S.D.
      • Portenoy R.K.
      Opioids for chronic noncancer pain: Prediction and identification of aberrant drug-related behaviors: A review of the evidence for an American Pain Society and American Academy of Pain Medicine clinical practice guideline.
      • Turk D.C.
      • Swanson K.S.
      • Gatchel R.J.
      Predicting opioid misuse by chronic pain patients: A systematic review and literature synthesis.
      It remains unclear whether it is possible to identify truly low-risk patients among those considering long-term use of opioids.
      Limitations of our methods include the use of data from only 1 health plan. Previous research on GHC COT patients has generalized to other settings
      • Dunn K.M.
      • Saunders K.W.
      • Rutter C.M.
      • Banta-Green C.J.
      • Merrill J.O.
      • Sullivan M.D.
      • Weisner C.M.
      • Silverberg M.J.
      • Campbell C.I.
      • Psaty B.
      • Von Korff M.
      Opioid prescriptions for chronic pain and overdose: A cohort study.
      • Saunders K.W.
      • Von Korff M.
      • Campbell C.I.
      • Banta-Green C.J.
      • Sullivan M.D.
      • Merrill J.O.
      • Weisner C.
      Concurrent use of alcohol and sedatives among persons prescribed chronic opioid therapy: Prevalence and risk factors.
      • Von Korff M.
      • Merrill J.O.
      • Rutter C.M.
      • Sullivan M.
      • Campbell C.I.
      • Weisner C.
      Time-scheduled vs. pain-contingent opioid dosing in chronic opioid therapy.
      • Von Korff M.
      • Ormel J.
      • Keefe F.J.
      • Dworkin S.F.
      Grading the severity of chronic pain.
      ; however, future research should replicate the prediction methods developed in this paper in other settings. With the growing use of EHR systems in diverse community practice settings, replication of this research in other clinical settings is possible. GHC instituted additional care processes for the management of prescription opioid use in its population toward the latter part of 2010.
      • Trescott C.E.
      • Beck R.M.
      • Seelig M.D.
      • Von Korff M.R.
      Group Health's initiative to avert opioid misuse and overdose among patients with chronic noncancer pain.
      Whether these changes influenced how clinicians observed and documented problem opioid use in the EHR after 2010 is not known at this time. We did not include information from urine drug screens in the analysis because this information was not systematically collected for a large percentage of COT patients within the GHC system until after 2010.
      Our NLP classification of problem opioid use included instances of prescription opioid addiction, abuse, misuse, diversion, and/or overuse of opioids. At this time, it is not known whether reliable differentiation of different types of problem opioid use (eg, addiction vs overuse) is possible. This limitation does not diminish the potential value of improving problem opioid use risk assessment, regardless of type, in community practice.
      For practical reasons, notably the large number of clinical notes per patient, we were not able to manually validate the absence of problem opioid use in the entire electronic chart of study subjects. It is possible that clinical notes not manually reviewed contained information pertaining to problem opioid use. To the extent that problem opioid use was unrecognized and/or undocumented by clinicians in this setting, ascertainment of problem opioid use was incomplete. Previous literature has also shown that problem opioid use is often underrecognized and underdiagnosed.
      • Krebs E.E.
      • Ramsey D.
      • Miloshoff J.M.
      • Bair M.J.
      Primary care monitoring of long-term opioid therapy among veterans with chronic pain.
      As such, our estimates of the risk for developing problem opioid use likely underestimate risks for problem opioid use in clinical practice.
      In terms of strengths, this study is one of the few longitudinal analyses of problem opioid use risk assessment, with by far the longest duration of follow-up. We assessed study subjects for 2 years prior to initiation of chronic opioid use and up to 5 years thereafter. Second, our results complement those of earlier predictive models that used only ICD-9 diagnoses from medical claims data to identify problem opioid use.
      • Dufour R.
      • Joshi A.V.
      • Pasquale M.K.
      • Schaaf D.
      • Mardekian J.
      • Andrews G.A.
      • Patel N.
      The prevalence of diagnosed opioid abuse in commercial and Medicare managed care populations.
      • Rice J.B.
      • White A.G.
      • Birnbaum H.G.
      • Schiller M.
      • Brown D.A.
      • Roland C.L.
      A model to identify patients at risk for prescription opioid abuse, dependence, and misuse.
      • Sullivan M.D.
      • Edlund M.J.
      • Fan M.Y.
      • Devries A.
      • Brennan Braden J.
      • Martin B.C.
      Risks for possible and probable opioid misuse among recipients of chronic opioid therapy in commercial and medicaid insurance plans: The TROUP Study.
      Third, our method offers busy, time- and resource-constrained health plans an opportunity to more efficiently implement automated surveillance of risk indicators for problem opioid use within their patient populations. The weighted count of risk indicators may be more readily implemented in community practice settings than use of a regression-based prediction model. This can be done by applying the weights we generated from our analysis to the values of the respective set of baseline risk indicators for each patient to generate a weighted count measure for each patient. Our results suggest that it may be possible to improve the efficiency of developing information relevant to assessing risk for problem opioid use among patients initiating COT.
      Clinical guidelines for chronic pain recommend that opioids be considered only after an adequate trial of nonopioid options, yet accurately assessing risk for problem opioid use remains a challenge.
      • Brown J.
      • Setnik B.
      • Lee K.
      • Wase L.
      • Roland C.L.
      • Cleveland J.M.
      • Siegal S.
      • Katz N.
      Assessment, stratification, and monitoring of the risk for prescription opioid misuse and abuse in the primary care setting.
      Washington State Agency Medical Directors’ Group
      Interagency Guideline on Opioid Dosing for Chronic Non-cancer Pain: An Educational Aid to Improve Care and Safety With Opioid Treatment.
      US Food and Drug Administration
      letter to Extended Release/Long Acting opioid application holders.
      Steps to improve timely identification of risk indicators for problem opioid use could play a role in reducing adverse selection of higher risk patients into COT; however, the effectiveness of recommended risk mitigation strategies has not been determined. Because predictive models for problem opioid use are only moderately accurate, at best, there is always a need for the clinician's vigilance to ensure safe and appropriate opioid use for long-term management of chronic musculoskeletal pain.

      Acknowledgment

      The authors would like to recognize the contributions of Patty Yarbro and Megan Addis and research specialist Chrystel Kratohvil of Group Health Research Institute and Annmarie Gillespie of Pfizer Inc for administrative support and of Sean Donevan of Pfizer Inc for medical and scientific input during the project.

      References

        • Birnbaum H.G.
        • White A.G.
        • Schiller M.
        • Waldman T.
        • Cleveland J.M.
        • Roland C.L.
        Societal costs of prescription opioid abuse, dependence, and misuse in the United States.
        Pain Med. 2011; 12: 657-667
        • Brown J.
        • Setnik B.
        • Lee K.
        • Wase L.
        • Roland C.L.
        • Cleveland J.M.
        • Siegal S.
        • Katz N.
        Assessment, stratification, and monitoring of the risk for prescription opioid misuse and abuse in the primary care setting.
        J Opioid Manag. 2011; 7: 467-483
        • Chou R.
        • Fanciullo G.J.
        • Fine P.G.
        • Adler J.A.
        • Ballantyne J.C.
        • Davies P.
        • Donovan M.I.
        • Fishbain D.A.
        • Foley K.M.
        • Fudin J.
        • Gilson A.M.
        • Kelter A.
        • Mauskop A.
        • O'Connor P.G.
        • Passik S.D.
        • Pasternak G.W.
        • Portenoy R.K.
        • Rich B.A.
        • Roberts R.G.
        • Todd K.H.
        • Miaskowski C.
        • American Pain Society-American Academy of Pain Medicine Opioids Guidelines Panel. Opioid Treatment Guidelines
        Clinical guidelines for the use of chronic opioid therapy in chronic noncancer pain.
        J Pain. 2009; 10: 113-130
        • Chou R.
        • Fanciullo G.J.
        • Fine P.G.
        • Miaskowski C.
        • Passik S.D.
        • Portenoy R.K.
        Opioids for chronic noncancer pain: Prediction and identification of aberrant drug-related behaviors: A review of the evidence for an American Pain Society and American Academy of Pain Medicine clinical practice guideline.
        J Pain. 2009; 10: 131-146
        • Cronkite D.
        Estimating Prevalence of Clinician-Labeled Opioid Abuse: Structured vs. Unstructured Clinical Data. HMO Research Network Annual Meeting Presentation.
        2014 (Phoenix, AZ)
        • Dufour R.
        • Joshi A.V.
        • Pasquale M.K.
        • Schaaf D.
        • Mardekian J.
        • Andrews G.A.
        • Patel N.
        The prevalence of diagnosed opioid abuse in commercial and Medicare managed care populations.
        Pain Pract. 2014; 14: E106-E115
        • Dunn K.M.
        • Saunders K.W.
        • Rutter C.M.
        • Banta-Green C.J.
        • Merrill J.O.
        • Sullivan M.D.
        • Weisner C.M.
        • Silverberg M.J.
        • Campbell C.I.
        • Psaty B.
        • Von Korff M.
        Opioid prescriptions for chronic pain and overdose: A cohort study.
        Ann Intern Med. 2010; 152: 85-92
        • Frakt A.
        Painkiller abuse, a cyclical challenge. New York Times.
        2014 (Available at:) (Accessed December 29, 2014)
        • Gourlay D.L.
        • Heit H.A.
        • Almahrezi A.
        Universal precautions in pain medicine: A rational approach to the treatment of chronic pain.
        Pain Med. 2005; 6: 107-112
        • Washington State Agency Medical Directors’ Group
        Interagency Guideline on Opioid Dosing for Chronic Non-cancer Pain: An Educational Aid to Improve Care and Safety With Opioid Treatment.
        Washington State Department of Labor and Industries, Olympia (WA)2010 (Available at:) (Accessed December 28, 2014)
      1. FDA Statement. FDA Commissioner Margaret A Hamburg Statement on Prescription Opioid Abuse. 2014 (Available at:) (Accessed April 28, 2014)
        • Institute of Medicine of the National Academies
        Relieving Pain in America: A Blueprint for Transforming Prevention, Care, Education, and Research.
        National Academies Press, Washington, DC2011
        • Jones T.
        • Moore T.
        • Levy J.L.
        • Daffron S.
        • Browder J.H.
        • Allen L.
        • Passik S.D.
        A comparison of various risk screening methods in predicting discharge from opioid treatment.
        Clin J Pain. 2012; 28: 93-100
        • Krebs E.E.
        • Ramsey D.
        • Miloshoff J.M.
        • Bair M.J.
        Primary care monitoring of long-term opioid therapy among veterans with chronic pain.
        Pain Med. 2011; 12: 740-746
        • Meier B.
        FDA urging a tighter rein on painkillers.
        2013 (Available at:) (Accessed March 30, 2014)
        • Moore T.M.
        • Jones T.
        • Browder J.H.
        • Daffron S.
        • Passik S.D.
        A comparison of common screening methods for predicting aberrant drug-related behavior among patients receiving opioids for chronic pain management.
        Pain Med. 2009; 10: 1426-1433
        • National Institute of Drug Abuse
        Prescription Drug Abuse.
        (Available at:) (Accessed March 31, 2014)
        • Office of National Drug Control Policy
        Epidemic: Responding to America’s Prescription Drug Abuse Crisis.
        2011 ([updated 2011 Apr 11; cited 2011 Jun 10]. Available at:) (Accessed December 28, 2014)
        • Passik S.D.
        • Kirsh K.L.
        • Casper D.
        Addiction-related assessment tools and pain management: Instruments for screening, treatment planning, and monitoring compliance.
        Pain Med. 2008; 9: S145-S166
      2. Prescription Drug Monitoring Program (PDMP) Center of Excellence, Brandeis University. National Rx Drug Abuse Summit Meeting, 22 April 2014

        • Rice J.B.
        • White A.G.
        • Birnbaum H.G.
        • Schiller M.
        • Brown D.A.
        • Roland C.L.
        A model to identify patients at risk for prescription opioid abuse, dependence, and misuse.
        Pain Med. 2012; 13: 1162-1173
        • Rolfs R.T.
        • Johnson E.
        • Williams N.J.
        • Sundwall D.N.
        Utah clinical guidelines on prescribing opioids for treatment of pain.
        J Pain Pall Care Pharm. 2010; 24: 219-235
        • Saunders K.W.
        • Von Korff M.
        • Campbell C.I.
        • Banta-Green C.J.
        • Sullivan M.D.
        • Merrill J.O.
        • Weisner C.
        Concurrent use of alcohol and sedatives among persons prescribed chronic opioid therapy: Prevalence and risk factors.
        J Pain. 2012; 13: 266-275
        • Savova G.K.
        • Masanz J.J.
        • Ogren P.V.
        • Zheng J.
        • Sohn S.
        • Kipper-Schuler K.C.
        • Chute C.G.
        Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): Architecture, component evaluation and applications.
        J Am Med Inform Assoc. 2010; 17: 507-513
        • Sehgal N.
        • Manchikanti L.
        • Smith H.S.
        Prescription opioid abuse in chronic pain: A review of opioid abuse predictors and strategies to curb opioid abuse.
        Pain Physician. 2012; 15: ES67-ES92
        • Starrels J.L.
        • Becker W.C.
        • Alford D.P.
        • Kapoor A.
        • Williams A.R.
        • Turner B.
        Systematic review: Treatment agreements and urine drug testing to reduce opioid misuse in patients with chronic pain.
        Ann Intern Med. 2010; 152: 712-720
        • Sullivan M.D.
        • Edlund M.J.
        • Fan M.Y.
        • Devries A.
        • Brennan Braden J.
        • Martin B.C.
        Risks for possible and probable opioid misuse among recipients of chronic opioid therapy in commercial and medicaid insurance plans: The TROUP Study.
        Pain. 2010; 50: 332-339
        • Trescott C.E.
        • Beck R.M.
        • Seelig M.D.
        • Von Korff M.R.
        Group Health's initiative to avert opioid misuse and overdose among patients with chronic noncancer pain.
        Health Aff. 2011; 30: 1420-1424
        • Turk D.C.
        • Swanson K.S.
        • Gatchel R.J.
        Predicting opioid misuse by chronic pain patients: A systematic review and literature synthesis.
        Clin J Pain. 2008; 24: 497-508
        • US Food and Drug Administration
        letter to Extended Release/Long Acting opioid application holders.
        2013 (Available at:) (Accessed July 10, 2014)
        • Von Korff M.
        • Kolodny A.
        • Deyo R.A.
        • Chou R.
        Long-term opioid therapy reconsidered.
        Ann Intern Med. 2011; 155: 325-328
        • Von Korff M.
        • Merrill J.O.
        • Rutter C.M.
        • Sullivan M.
        • Campbell C.I.
        • Weisner C.
        Time-scheduled vs. pain-contingent opioid dosing in chronic opioid therapy.
        Pain. 2011; 152: 1256-1262
        • Von Korff M.
        • Ormel J.
        • Keefe F.J.
        • Dworkin S.F.
        Grading the severity of chronic pain.
        Pain. 1992; 50: 133-149
        • White A.G.
        • Birnbaum H.G.
        • Mareva M.N.
        • Daher M.
        • Vallow S.
        • Schein J.
        • Katz N.
        Direct costs of opioid abuse in an insured population in the United States.
        J Manag Care Pharm. 2005; 11: 469-479
        • Wilson P.
        • D’Agostino R.
        • Levy D.
        • Belanger A.
        • Silbershatz H.
        • Kannel W.
        Prediction of coronary heart disease using risk factor categories.
        Circulation. 1998; 97: 1837-1847