Abstract
Background Antidepressant medication adherence is among the most important problems in health care worldwide. Interventions designed to increase adherence have largely failed, pointing towards a critical need to better understand the underlying decision-making processes that contribute to adherence. A computational decision-making model that integrates empirical data with a fundamental action selection principle could be pragmatically useful in 1) making individual level predictions about adherence, and 2) providing an explanatory framework that improves our understanding of non-adherence.
Methods Here we formulate a partially observable Markov decision process model based on the active inference framework that can simulate several processes that plausibly influence adherence decisions.
Results Using model simulations of the day-to-day decisions to take a prescribed selective serotonin reuptake inhibitor (SSRI), we show that several distinct parameters in the model can influence adherence decisions in predictable ways. These parameters include differences in policy depth (i.e., how far into the future one considers when deciding), decision uncertainty, beliefs about the predictability (stochasticity) of symptoms, beliefs about the magnitude and time course of symptom reductions and side effects, and the strength of medication-taking habits that one has acquired.
Conclusions Clarifying these influential factors will be an important first step toward empirically determining which are contributing to non-adherence to antidepressants in individual patients. The model can also be seamlessly extended to simulate adherence to other medications (by incorporating the known symptom reduction and side effect trajectories of those medications), with the potential promise of identifying which medications may be best suited for different patients.
Introduction
Medical treatment depends crucially on a patient’s decision to adhere. Unfortunately, the number of patients who follow treatment recommendations is quite low (1). Nearly half of prescribed medications are not taken and roughly 125,000 deaths each year are due to non-adherence; costs associated with non-adherence are between $100 and $300 billion annually (2). In the context of mental healthcare – the focus of this paper – roughly one in five patients adhere to antidepressant medication treatment for more than four months (3), and the majority discontinue within 30 days (4). Low adherence rates do not appear fully attributable to an objective lack of efficacy, as those who follow treatment recommendations show lower recurrence risk (5), cardiovascular mortality (6), overall mortality (7, 8) and lower suicide rates (9). Individuals who adhere also report greater perceived benefits and fewer concerns than those who do not (1). Thus, it is somewhat perplexing that patients often do not adhere.
Beliefs about medication and personality attributes both influence adherence. Adherence to psychiatric interventions is lower in those with lower treatment expectancy (10) and in those who experience sexual side effects (11). Such findings have led to the ‘necessity-concerns framework’ (12) which proposes that adherence decisions involve weighing expected negative outcomes against beliefs about the necessity/efficacy of treatment (13–18). Personality variables associated with greater adherence include higher persistence (19, 20), greater self-efficacy (21–23), lower optimism (24), greater self-control (25), and greater internal locus of control (26)(27).
Habit formation may also be central to the development of stable health behaviors (28), an effect that could generalize to adherence behavior (29, 30). However, studies have found that the amount of time required to form strong habits is highly variable (e.g., 18 to 254 days in one study (31)). The processes that moderate habit formation time are therefore also relevant to understanding long-term adherence.
While several interventions to promote adherence have been tested, involving educational (32, 33) counseling (34), and coaching (35) approaches, they have shown low efficacy in randomized controlled trials (36). There is therefore a need to better understand the decision-making processes contributing to adherence. A precise and quantitative delineation of these processes could help inform and develop targeted interventions focused on specific adherence-related processes and provide better measures for predicting adherence.
Computational psychiatry approaches have recently gained prominence, due in part to their potential to quantitatively model behavior and illustrate how several underlying processes can contribute to maladaptive perception and decision-making (37–40). In this manuscript we introduce a computational model of antidepressant medication adherence, based on the active inference framework (41), and examine how it might provide a detailed characterization of different probabilistic beliefs and related inference processes that plausibly contribute to adherence decisions. This approach could help clarify the way individual differences in computational processes can arbitrate between adherence and non-adherence in individual patients. We will focus on the initial decision to adhere over the first 12 weeks of treatment, in which symptoms typically decline and stabilize (42) and in which side effects first appear and subsequently reduce somewhat in severity (43). We also simulate processes that may moderate the speed with which individuals develop strong medication-taking habits after their initial decision to adhere.
Method
An active inference model of adherence
The active inference framework (41) is one approach to provide a computational framework for adherence. This framework assumes the brain develops a generative model that represents different possible states of the world and generates predictions about the outcomes it will observe if its beliefs are accurate. It uses observations to update the model and to generate sequences of actions (policies) that minimize a statistical quantity called ‘expected free energy’ (G); briefly, policies that minimize G are those expected to produce the most preferred (rewarding) observations while also maximizing information gain (see table 2). The specific model we use is a Markov decision process (MDP; figure 1), which is particularly useful for modeling decision processes that require consideration of distal future outcomes under uncertainty (44, 45). The formal basis for these models has been thoroughly detailed elsewhere (41, 46–49) (also see tables 1 and 2 for more mathematical detail).
One approach to examine the computational processes that might affect adherence is to model the decision-making situation for an individual that is newly prescribed an antidepressant for a depressive episode (see figure 2). We selected a time frame of the first 12 weeks since most antidepressant trials focus on this period. Moreover, we utilized previously reported decreases in depressive severity (42) and the temporal emergence of unwanted effects (43). In our model, the policy to adhere or not on (any day during) each of 12 weeks of treatment (13 policies total, formally modeled as one choice per week) was based on initial expectations about how depressive symptom severity (from 0-10) and side effect severity (from 0-3) would change over time given each policy, and on how these expectations were subsequently updated when changes in symptoms and side effects were observed by the simulated patient. The 2 state factors in our model correspond to beliefs about “progress in treatment” (which predict different combinations of observed symptom and side effect levels), and beliefs about whether one is currently choosing to adhere (which predict different patterns of change over time in symptom and side effect levels).
Several matrices/vectors in an MDP define the probabilistic relationships between each of these variables (see figure 3A); these include matrices encoding the relationships between states and observations (A-matrices; 1 per observation modality), how states are expected to change over time (B-matrices; at least 1 per state factor – if greater than 1, each possible transition corresponds to an action option), the relative preference for some observations over others (C-matrices; 1 per observation modality), expectations about the initial states one will start out in (D-vectors; 1 per state factor), and prior expectations about which policies one is most likely to choose in general (E-vector; 1 probability entry assigned to each policy). For more mathematical detail about each of these variables/matrices, see tables 1 and 2 and figure 1.
The A-matrices were constructed such that steadily lowering (but fluctuating) symptom levels were generated for each week the patient continued to take the medication. Specifically, Gaussian distributions were specified over a symptom severity scale from 0 to 10, with empirically based means that probabilistically decreased from baseline levels over the 1st 6 weeks of treatment and then remained somewhat stable thereafter (i.e., depending on the standard deviations set for these distributions). The means were based on dose response time courses characterized in a mega analysis of three SSRIs (42): 9.36, 8.04, 6.91, 5.94, 5.10, 4.38, 3.77. Adherence also generated moderate side effect levels in the first six weeks and mild side effect levels from weeks 7-12 (based on the antidepressant side effect time courses characterized in (43)). The choice to cease adherence led to the cessation of side effects and a gradual return to baseline symptom levels.
The B-matrices were constructed so that the patient controlled the transition from the state of ‘continue medication’ to the state of ‘discontinue medication’ at each week (i.e., if two individuals ceased taking the medication on different days during the same week, these were formally treated as the same choice). Simultaneously, each action was associated with transitions toward increasing or decreasing “progress in treatment” levels, respectively. The C-matrices were constructed such that the patient most preferred the lowest symptom levels (ln P(o): from − 10 to 0) and the lowest side effect severity (ln P(o): 0, −2, −4, −6). The D-vectors were constructed such that the patient always began in an initial state of being “undecided” about choosing to adhere, and always began in an initial state of being at 0 weeks of treatment progress. The E-vector was initially set such that the patient had no bias (habit) for choosing to adhere or not (i.e., a flat prior distribution over policies).
The parameters in our model are described in table 3, and their influence on the matrices/vectors defining the model are depicted in figure 3B. These correspond to individual differences in (beliefs about) the predictability and magnitude of changes in symptoms and side effects, how far into the future one considers when making decisions (policy depth), confidence in the consequences of choosing different actions (expected policy precision), and the strength of medication-taking habits.
Results
Simulating individual differences in adherence
Initial simulations
The left panel of figure 4 illustrates an example simulation under one set of parameter values, in which policy depth was at its maximum value of 12, symptom predictability was moderate (SD = 2), symptom reductions were high (i.e., the simulated patient was a “strong responder” to the medication), side effects were moderate, expected policy precision was high (β = 1), and no medication-taking habit had been formed (the E-vector distribution remained flat over all policies). Here the patient chose to adhere and observed steady symptom reductions and some initial moderate side effects that decreased to mild levels after the first six weeks. The middle panel illustrates another example simulation in which symptom predictability was moderate (SD = 2), symptom reductions were moderate, and side effects were severe. In this case, the patient chose not to adhere. In these simulations, the patient’s expectations were consistent with subsequent observations. The right panel illustrates a simulation in which the patient believed symptom predictability was higher than it really was (SD = 0.5 vs. 2) and side effects and symptom reductions were moderate (which was expected). The patient took the medication the first two weeks, but ceased adhering after observing an unexpected increase in symptom levels.
Parameter interactions in the context of accurate expectations
To better characterize this parameter space, we repeated the simulations above under many combinations of parameter values. Figure 5 illustrates two example plots from part of this parameter space where the patient had accurate expectations; medication response magnitude and side effect severity were fixed at moderate levels. Here, the x-axis in each plot corresponds to symptom predictability (from low to high: SDs from 4 to 0.1); the y-axis corresponds to policy depth (from 2 to 12 weeks). The left plot corresponds low expected policy precision (β = 10), and the right corresponds to high expected policy precision (β = 1). Black squares correspond to patients that remained adherent, whereas white squares indicate ceased adherence prior to week 12. As can be seen, there is a clear boundary in which, below a certain policy depth and level of symptom predictability, adherence ceases. Interestingly, adherence increases across the space when expected policy precision is low. This suggests that, with greater decision uncertainty (it is less clear that one policy will produce more preferred observations than others), one is more likely to adhere in an “exploratory” manner (e.g., “I don’t think this will work, but who knows?”).
Figure 6 provides a more complete depiction of this parameter space under different levels of medication response magnitudes (larger x-axis across plots, from very weak to strong response) and side effect severities (larger y-axis across plots, from low to high severity). Under high side effect severity and low response magnitude, adherence does not occur no matter the policy depth or symptom predictability (upper left plot), whereas adherence occurs broadly when side effects are low and response magnitude is high (bottom left plot).
Parameter interactions in the context of inaccurate expectations
Crucially, in the previous simulations it was assumed that the patient’s beliefs about the effects of medication were accurate. However, a patient’s beliefs need not match reality. As such, we then dissociated the patient’s beliefs from the statistics of subsequent observations to see if surprise would influence adherence in interesting ways. Supplementary figure S1 depicts some illustrative locations in this space. The x-axis in each graph corresponds to subjective beliefs about symptom predictability, where actual symptom predictability instead varies between groups of plots (see figure legend for more details). Similarly, beliefs about drug response magnitude and side effect severity are now depicted across plots within each group of plots, whereas actual drug response magnitude and side effect severity vary across groups of plots.
When observed symptom reductions were highly unreliable/noisy, the patient only remained adherent if prior expectations very precisely predicted that they would be reliable. When observed symptom reductions were instead highly consistent/reliable, the patient surprisingly also chose to adhere when expecting symptom reductions to be highly unreliable. Thorough inspections of the parameter space confirmed that, in these cases of highly reliable or unreliable symptom reductions, there was little influence of expected policy precision, drug response magnitude, or side effect severity. Other interesting results were observable in cases of objectively moderate levels of symptom reduction reliability. For example, there appears to be a general effect in which adherence was higher when expectations about side effects matched the actual side effects observed. This effect was more pronounced in the case of objectively strong drug responses; in the case of objectively weak drug responses and severe side effects, adherence levels became more dependent on the expected reliability of symptom reductions. In cases of objectively low side effects, adherence went up in general, most notably in cases of high policy depth and the belief that symptom reductions would be unreliable (unless expected side effects were severe).
The finding that adherence is higher when expected observations about side effects are confirmed appears sensible, in that, if one initially chose to “try out” adherence based on one’s expectations, non-preferred surprising observations would be more likely to “change one’s mind” later. The finding that, in the context of objectively reliable symptom reductions, adherence occurs at low but not intermediate levels of expected symptom predictability is initially more surprising. However, when symptoms are expected to fluctuate very unreliably, it makes sense retrospectively that being “pleasantly surprised” that observations are more reliable than expected would promote adherence. In contrast, if one initially expects moderate predictability, the level of surprise may not be sufficient to change one’s mind. Finally, it is fairly intuitive that, in the face of highly fluctuating symptoms, adherence would require a strong counteracting belief that they would still be reliable at future time points (e.g., “it’s been a bumpy ride to start, but I think things are going to stabilize soon”).
Parameter interactions in the context of habit formation
To investigate why some individuals take much longer to form strong habits than others (as reviewed in the introduction), we ran a final set of simulations in which we manipulated the strength of the patient’s habit to take medication. We manipulated the E-matrix to simulate the effect of a patient having previously taken the medication different numbers of times (i.e., 1, 15, 30, and 60 previous medication-taking decisions). Policy depth was set to 1, such that the patient was not forward-looking beyond the immediate expected consequences of adhering. As can be seen in figure 7, different simulated patients begin to adhere habitually after different lengths of time depending on other parameter values. Longer time periods of previous adherence lead to habitual adherence. Habits took longer to develop when expected drug responses were low and unpredictable, and when expected side effect severity was high (i.e., the medication-taking “impulse” less effectively competed against explicit planning when strongly non-preferred or unpredictable outcomes were expected to occur immediately). Low expected policy precision led to much faster habit formation. Psychologically, this might be interpreted as indicating an interesting (and somewhat paradoxical) predicted effect. That is, in the context of higher side effects and lower drug responses, individuals who are less confident in decision-making should be more likely to adhere long-term than those who confidently predict at treatment onset that the drug will not be very helpful.
Discussion
The goal of this manuscript was to provide a computational framework to develop novel approaches that can be used to increase adherence. Simulations were used to quantitatively illustrate distinct decision-making processes that could contribute to differences in adherence behavior. These simulations demonstrate how adherence can be influenced by several underlying computational processes. This supports the plausibility of heterogeneous causes of non-adherence (for a summary of possible causes in our model, see table 3). For example, while some patients may focus on proximal as opposed to distal future outcomes (i.e., low policy depth), others may focus on distal future outcomes but believe those outcomes are highly unreliable or that they will be worse overall if they follow treatment recommendations. Some patients may be characterized by intermediate combinations, such as intermediate policy depth and competing beliefs about moderately beneficial medication effects and moderately aversive side effects, both weighted by the relative confidence they have in each of those beliefs. Finally, differences in previous experience taking medication can lead to differences in adherence-promoting habit formation – where each of the other factors described above can influence how quickly such habits gain sufficient strength to maintain long-term adherence behavior. One advantage of the model is that each of these factors and their interactions can be simulated mathematically. The pragmatic utility of this approach will depend on developing experimental paradigms and/or questionnaires to estimate parameter values based on a given individual’s behavior. This model could potentially make predictions about where interesting “tipping points” would begin to favor non-adherence over adherence in that individual’s decision process.
At least three important research implications follow. First, it will be important to either identify or design simple measures (e.g., that could be administered in a clinic) capable of characterizing where an individual patient falls within the parameter space we have described. Such measures could potentially improve prediction of who will and will not adhere to a prescribed treatment. For example, our model would predict lower adherence in those who self-report overconfidence in their current plan of action, the tendency to focus on short-term outcomes, and either pessimistic or unrealistic expectations about how their symptoms and side effects would change over time.
With respect to existing measures, the values for some parameters might be assessed by those used within the “necessity versus concerns” framework, such as the Beliefs about Medicines Questionnaire (12) or the revised Medication Adherence Reasons Scale (50) – perhaps most plausibly the parameters associated with beliefs about the magnitude of symptom reductions and side effects. In table 4, we have also listed a number of example self-report items that, based on our model (and straightforward extensions of it), could be useful for gathering information about a patient’s adherence-relevant beliefs. Once in a validated form, this type of questionnaire could be tested for its utility in predicting adherence behavior in advance.
The values of other parameters, in contrast, might be informed by personality measures. For example, the behavior associated with persistence (20) and locus of control (26) could follow from a combination of beliefs in high future predictability and high expected policy precision (and likely preferences with strong magnitudes as well). Low self-efficacy (21) could instead follow from a combination of low expected policy precision and precise expectations for non-preferred outcomes (while optimism may correspond to precise expectations for preferred outcomes, perhaps with or without taking medication – which could explain why more optimistic individuals are less likely to adhere; (24)). This is speculative, however, and will need to be examined in future work.
A second implication is that it will be important within the field of computational psychiatry to attempt to develop tasks, or find ways of acquiring detailed adherence behavior data, that could be used in conjunction with a formal model (such as the one we described above) to explicitly fit to patient behavior. This would potentially allow for a more detailed and informative way to phenotype decision-making processes within individual patients and afford more precise empirical predictions. It should be mentioned, however, that this may be challenging since the behavior in question involves simple binary decisions. If multiple parameter value combinations can produce the exact same behavior, this prevents identification of unique parameter values that best explain individual patients’ behavior. As such, this endeavor will likely require use of very detailed behavioral data, perhaps involving day-by-day medication-taking actions.
A third implication pertains to the need for effective adherence-promoting interventions. As reviewed above, current interventions have met with limited success (36), which could be due to heterogeneity in underlying mechanisms as well as a failure to specifically target those mechanisms. Based on the factors highlighted in our model, it might be possible to improve the ability of current interventions (or design new interventions) to intervene in a targeted manner. The degree of modifiability in those factors is an open question, but certainly one worth pursuing. Based on our simulations, we would predict that both of the following should improve adherence:
Increasing future-oriented thinking (policy depth)
Attenuating overconfidence (expected policy precision)
Additionally, we would predict that adherence would be improved by first providing patients with available information about what to expect regarding likely symptom and side effect trajectories (and the actual uncertainty around those expectations) – and then using observations early in treatment to update expectations about the most likely trajectory for that individual. This goes beyond simple psychoeducation, which alone has been found to be ineffective in improving antidepressant adherence rates (51), and should motivate future research into how individual symptom and side effect trajectories could be diagnosed as early as possible. This also relates to potential uses of our model for psychoeducation. For example, by making the various model components salient to a patient (e.g., by considering their own answers to the example questions in table 4), they could become more aware of how these beliefs influence their own decision-making.
It is important to stress the oversimplified and incomplete nature of the model we have presented. For simulations one must unavoidably make somewhat arbitrary decisions about the values that should be assigned to fixed parameters (and what parameters to fix). For example, we chose to set the preference magnitudes for side effects and symptom levels to specific values, and the simulation results would be expected to differ somewhat if different values had been chosen. A possible future direction could be to calibrate the model to individuals’ preference magnitudes (e.g., how much they personally dislike particular symptoms and side effects) and use this to simulate/predict their future adherence behavior. We also did not manipulate beliefs about the predictability of side effects, which could influence adherence decisions. In these cases, we instead chose to hold these factors fixed and examine the effect of altering the dynamics of observed symptoms/side effect levels over time on versus off treatment.
There may also be additional factors that were not modeled explicitly, but that could be simulated within the model we have presented (and in principle used in characterizing patients’ decisions). For example, some individuals may simply forget to take their medication a couple of times and then cease altogether (50). In our model, this could be captured in part by low expected policy precision (i.e., increasing randomness in behavior), but would need to be combined with other factors such as the belief that all progress has been lost (as could be encoded within the individual’s transition beliefs [B-matrix]). As another example, some people might cease medication after experiencing symptom improvement because they expect such improvements to remain stable after ceasing to take the drug (and the belief that side effects would go away (52)). This could also be captured in our model with straightforward adjustments to the B-matrix.
Conclusion
This active inference model is an important first step in developing a precise and quantitative delineation of decision-making factors and dynamics that influence a patient’s decision to adhere to treatment. Because of the model’s generality, it can also be very easily extended to model adherence to other medications, simply by inserting the symptom reduction and side effect profiles that characterize those medications. For example, it follows from the general model structure that adherence to immediately rewarding medications (e.g. benzodiazepines) would be high and promote fast habit formation, but behavior would also be influenced by the same parameters used here to investigate antidepressant adherence. The next steps in using these models will require identifying means of empirically characterizing and intervening on these mechanisms in an individualized manner.
Software note
Although the generative model – specified by the various matrices described in this paper – changes from application to application, the belief updates are generic and can be implemented using standard routines (here spm_MDP_VB_X.m). These routines are available as Matlab code in the DEM toolbox of the most recent versions of SPM academic software: http://www.fil.ion.ucl.ac.uk/spm/. The simulations in this paper can be reproduced via running the Matlab code included here as supplementary material (adherence_model.m).
Financial Disclosure
All authors reported no biomedical financial interests or potential conflicts of interest.
Acknowledgments
This work has been supported in part by The William K. Warren Foundation, the National Institute of Mental Health Award Numbers K23MH112949 (SSK), and the National Institute of General Medical Sciences Center Grant Award Number 1P20GM121312 (MPP and SSK). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The authors would like to acknowledge Michael Moutoussis for his helpful suggestions during revision of the manuscript.
Footnotes
Additional figures and other edits during review process