SupervisorBot: NLP-Annotated Real-Time Recommendations of Psychotherapy Treatment Strategies with Deep Reinforcement Learning

Abstract

We propose a recommendation system that suggests treatment strategies to a therapist during the psychotherapy session in real-time. Our system uses a turn-level rating mechanism that predicts the therapeutic outcome by computing a similarity score between the deep embedding of a scoring inventory, and the current sentence that the patient is speaking. The system automatically transcribes a continuous audio stream and separates it into turns of the patient and of the therapist using an online registration-free diarization method. The dialogue pairs along with their computed ratings are then fed into a deep reinforcement learning recommender where the sessions are treated as users and the topics are treated as items. Other than evaluating the empirical advantages of the core components on existing datasets, we demonstrate the effectiveness of this system in a web app.

1 Introduction

Mental illness is not only a severe healthcare problem in the US (1 in 5 estimated by National Institute of Mental Health) but also a major global issue Patel et al. (2018). However, most countries including the states face severe shortage of mental health practioners, such as psychiatrists and clincal psychologists Satiani et al. (2018). In recent two years, this demand gap was especially amplified by the toll of COVID-19 pandemic on everyone’s mental health Wang et al. (2020b). Current education systems and training programs cannot catch up to this trend because each licensed therapist requires years of continual learning and supervised training. Even when a therapist is ripe for independent practice, many still seek weekly supervision from "supervisors", who are usally a more senior therapist that have seen many more years of patients and serve as "a crucial triad of learning difficulties that tend to confront beginning therapists in their training" Watkins Jr (2013). These supervisors provide necessary guidance and periodic feedback to junior therapists with respect to their development of mindedness, psychotherapist identities and treatment roadblocks they face in their own cases.

Figure 1: Analytical Framework of SupervisorBot System. (Left) Major components of the recommendation systems. (Right) Reinforcement learning framework of the psychotherapy recommendation system problem.

Figure 2: Example inventory items and scales

Figure 3: Empirical results on MiniVox and AlexStreet datasets

Agent r DDPG 0.2712 TD3 0.2192 BCQ 0.2843

Figure 4: Evaluation of deep reinforcement learning recommenders. (Left three) loss functions for the sub-networks for DDPG, TD3, and BCQ. (Right) Pearson’s r of the actual actions taken in the test set with their predicted actions.

Figure 5: State screenshots of SupervisorBot web app: inventory inputs, diarization training, state annotation, strategy recommendation.

In this work, we present SupervisorBot, a virtual AI companion that provides real-time feedback and recommends treatment strategy to the therapists while they are conducting their own psychotherapy. Like a supervisor, SupervisorBot offers feedback and guidance that are case-dependent. Like a supervisor, SupervisorBot has seen thousands of historical therapy sessions and case studies.

The base of our recommendation system relies on a rating system that evaluates how good a treatment strategy is. As the mental state of a patient can be complicated to characterize, we gravitates our approach towards well-defined clinical outcomes. The working alliance is such a psychological concept that is shown to be highly predictive of the success of psychotherapy in clinical setting Wampold (2015). It describes several important cognitive and emotional components of the relationship between these two agents in conversation, including the agreement on the goals to be achieved and the tasks to be carried out, and the bond, trust and respect to be established over the course of the dialogue Bordin (1979). It measures the tendency for communication partners to align with each other both in their verbal and non-verbal behaviors. We developed a natural language processing (NLP) approach to infer this quantity in real-time as ratings.

Our recommendation system transcribes the session in real-time, predicts the therapeutic outcome as a turn-level rating, and recommends treatment strategy that is best for the current context and state of the psychotherapy. It is web-based, interactive, informative, and learns continually. We believe this system is the first step towards solving the global issue of mental health by augmenting the treatment and education of clinical practitioners with a recommendation system of therapeutic strategy.

2 Methods

Fig 1 is an outline of the analytic framework. The continuous audio stream is fed into the system. First, we perform the speaker diarization by training the system for a few rounds by interacting with sparse feedback from the user.

Online speaker diarization. In research setting, the existing diarization methods often start with a registration stage training deep learning models on a pool of speaker profiles. In real world setting, on the other hand, it is suboptimal to have to register voiceprints for all users before an abitrary multi-user zoom meeting as a deployed system. Instead, a better lightweight deployable speaker diarization system should register the user when necessary instead of relying on pre-registering all the users. In order to efficiently adaptive to complex environment, they should also be able to run without depending on large-scale data access and mock up cold-start new user profiles using prior knowledge. This assumption requires the model to learn how to label the user profiles from scratch by having the user to interacting with the system and provide feedback to correct the system on the fly. To met these criteria for our real-time system, we follow Lin and Zhang (2021, 2020) and use BerlinUCB Lin (2020), an online semi-supervised learning bandit algorithm to do diarization. It separates audio into dyads of doctor-patient, which are then transcribed into natural language turns for real-time downstream analyses.

Therapeutic quality ratings. After we obtain a relatively well diarization result, we can configure the quality assessment setting by specifying a proper inventory. In this demonstration, we use the Working Alliance Inventory (WAI), a set of self-report measurement questionnaire that quantifies the therapeutic bond, task agreement, and goal agreement Horvath (1981); Tracey and Kokotovic (1989); Martin et al. (2000). Operationally, our goal is to derive from these 36 items three alliance scales: the task scale, the bond scale and the goal scale. They measures the three major themes of psychotherapy outcomes: (1) the collaborative nature of the dialogue participants’ relationship; (2) the affective bond between them, and (3) their capabilities to agree on treatment-related short-term tasks and long-term goals. The score corresponding to the three scales comes from a key table (Fig 2) which specifies the positivity or the sign weight to be applied on the questionnaire answer when summing in the end. The full scale is simply the sum of the scores of the three scales. The key table is like a weighting matrix that specifies the directionalities of the scales.

Transcription and real-time rating assessment. Now we are ready for real-time quality annotation. Given the audio stream for a given user, we first transcribe the diarized audio stream with standard automatic speech recognition module Adorf (2013). Following the approach proposed in Lin et al. (2022b); Lin (2022b, f), we embed both the dialogue turns and the inventories with deep sentence or paragraph embeddings (in this case, the SentenceBERT Reimers and et. al. (2019)), and then compute the cosine similarity between the embedding vectors of the turn and its corresponding inventory vectors. With that, for each turn (either by patient or by therapist), we obtain a 36-dimension working alliance score, which we may save in a relational database as in Lin (2022c).

Topic modeling as recommendation items. First, we define the “items”, “users”, “contents” and “ratings” in our recommendation system. Here, the “items” the system recommends are treatment strategies. In this example, we represents these strategies as a topic that the therapist should initiate or continue for the next turn. Given a large text corpus of many psychotherapy sessions, as in Lin et al. (2022a) we can first perform topic modeling (e.g. Miao et al. (2016, 2017); Nan et al. (2019); Dieng et al. (2020); Wang et al. (2020a)) to extract the main concepts discussed in the psychotherapy. We use the Embedded Topic Model (ETM) Wang et al. (2020a) in this work because it was shown to create the most diverse concepts in psychological corpus Lin et al. (2022a). One can also adopt a symbolic approach to the topic modeling to gain further insights into the causalities and relationships between these topical concepts as in Lin (2022d). In this study, we use annotate each turn with their most likely topic and identifies seven unique topics (Topic 0 is about figuring out, self-discovery and reminiscence; Topic 1 is about play. Topic 2 is about anger, scare and sadness. Topic 3 is about counts. Topic 6 is about explicit ways to deal with stress, such as keep busying and reaching out for help. Topic 7 is about numbers. Topic 8 is about continuation and keep doing.)

Recommendation system setting. Then, we pair these “items” with the “users” and “contents”, which in our case, would be the patientID, his or her previous turns, their aggregated formats and other meta data. For instance, we know that within each sessions, there exists many pairs of turns, and they would belong to the same “user”. However, one can also assign all turns belong to one clinical label, or all turns related to a certain topic as one “user”. In this example, we choose the session ids as users. And lastly, the “ratings” would be patient’s inferred alliance scores predictive of the therapeutic outcomes. Creating this database from historical data, we can train our system. Since we have defined our users, items, contents and ratings, the recommendation engine can be easily crafted with content-based Pazzani and Billsus (2007); Basu et al. (1998); Aggarwal and others (2016) and collaborative filtering Sarwar et al. (2001); He et al. (2017); Koren et al. (2022); Su and Khoshgoftaar (2009). Since our session turns are sequential and can specify a state or timestamp, it might be suitable for reinforcement learning (RL) Zheng et al. (2018); Wang et al. (2014); Zou et al. (2020) and session-based approaches Li et al. (2017); Wu et al. (2019); Ludewig and Jannach (2018), which can be neuroscience or psychiatry-inspired Lin et al. (2019, 2020a, 2021, 2020b) to provide further interpretable clinical insights.

Deep reinforcement learning recommendation approaches. Reinforcement learning approaches are effectively applied in language and speech tasks (as reviewed in Lin (2022e)), among which recommendation is an important use case. As shown in the right panel of Figure 1, the reinforcement learning environment is formulated such that the recommendation agent takes an action by recommending a strategy (say, a discussion topic). And the therapist will interact with the patient taking that suggestion into account. The dialogue interaction, in turn, has a quality evaluation of some sort (say, the therapeutic working alliance score). This serves as a reward to the recommendation agent to update its weights. In the meanwhile, the state is progressed to the next therapeutic states. As a first step, we evaluate three popular deep RL algorithms. Based on the deterministic policy gradient in an actor-critic architecture, the Deep Deterministic Policy Gradients (DDPG) Lillicrap et al. (2015) is a model-free algorithm for continous action spaces, and one of the first successful algorithms to learn policies end-to-end. Building upon the Double Q-Learning Hasselt (2010), Twin Delayed DDPG (TD3) Fujimoto et al. (2018) is a similar solution is proposed to correct for the overestimated value issue, and yields more competitive results in various game settings. As the online data collection of RL models are usually time consuming, in real world industrial setting, these models are usually trained using previously collected data. As a result, there is a growing popularity of offline reinforcement learning methods Levine et al. (2020). Among them, Batch Constrained Q-Learning (BCQ) Fujimoto et al. (2019) is the first continuous control deep RL algorithm that yields competitive results in off policy evaluations by restricting the agent’s exploration in the action space.

Ethical considerations. Following the ethical guidelines in Matthews et al. (2017); Graham et al. (2019) and the operational suggestions in Lin (2022a), we make sure that all training examples that we evaluate on are properly anonymized with pre- and post-processing techniques, and disclaim that these investigations are proof of concept and require extensive caution to prevent from the pitfall of over-interpretation.

3 Operational validation of components

We validated the performance of the speaker diarization component in MiniVox benchmark Lin and Zhang (2021) and report a robust cumulative diarization accuracy at each time step (Fig 3A).

For the rating computation, since there are no ground truth, we analyze the Alex Street Psychotherapy dataset which consists of transcribed recordings of over 950 therapy sessions between multiple anonymized therapists and patients. We observe that our alliance scores significantly predict suicidality in the patients (Fig 3B) and produces interesting and interpretable trajectories during the therapy sessions of different psychiatric conditions in both the alliance space (Fig 3B) and the topic space (Fig 3C). We notice that the treatment strategy adopted by experienced therapists differs when facing patients of different disorders.

To evaluate the recommendation systems, we preprocess the Alex Street dataset into a recommendation format (219,999 recommendation actions) and then split it into 95/5 train-test sets. To set up the batch training for reinforcement learning, we cut the turns into frames of 10 turn pairs and a batch size of 32. We train the three agents each for 100 epochs, where their losses consistently drop and converge (Figure 4 left panels) in a stable way. To compare them, we compute the Pearson’s r of the recommended actions with their corresponding ground truth actions the test set. We observe that BCQ is the best performing model with a correlation of 0.2843, followed by DDPG (0.2712) and TD3 (0.2192). The slight advantage might be due to the additional errors in not-offline methods introduced by extrapolation. This evaluation provides a proof of concept. Future work will focus on systematically comparing a larger spectrum of deep reinforcement learning and model architectures.

4 Interactive System: SupervisorBot

“SupervisorBot” is an interactive web-based system (Fig 5). We first give users the instructions on how to use the system. Then they are lead to input their own inventory used to evaluate the dialogue quality. In this case, we put in a default one, using the working alliance inventory. They are guided to input the score scale corresponding to each inventory item and click on “Analyze” to finalize. In the speaker diarization part, we compute and visualize the Mel Frequency Cepstral Coefficients (MFCC) in a sliding window fashion given the real-time audio input from microphone, with the MFCC bands color coded in the page. Finishing these two steps as the preparation, the system is now running, and the therapist can sit back and go on with the session. The app now moves to the annotation panel, where the therapist can see that a transcript is displayed, along with who is speaking. The computed alliance score in the three scales are also dynamically displayed in real-time according to the content of the dialogue turn. This is helpful information to assist the therapist. And in our last panel, we have our recommendation guidance. The topics to choose from are ranked and top N are displayed. The therapist can use it as a hint and initiate his response given a top recommendation. The system will transcribe his response and highlight the topic he most likely ended up choosing in the last round, and save that information as part of historical data. The system refreshes its parameters at the end of each session to fit new data.

5 Conclusions

In this work, we provide a practical example of how a real-time recommendation system can help therapists better treat their patients in psychotherapy sessions with informative clinical annotations and recommendations of treatment strategies with deep reinforcement learning. Although in this example, the strategies are the topics for the therapist to initiate or continue, the same approach can be extended to more complex and nuanced treatment suggestions. For instance, in the ABC approach of cognitive behavioral therapy (CBT), our system can suggest a belief (B) to guide the patients to better understand the causality between the activating event (A) and its consequence (C).

Before we conclude, another interesting perspective to view this line of research is hidden in Figure 1: while the recommendation agent is driven by reinforcement learning, the therapist (and even patient) have their agency which updates under the reinforcement learning principles. For instance, the patient can directly offer feedback to the therapists. And given the feedback, the therapist may adjust his or her internal model to weigh on the quality of the suggestions by the recommendation agent. Next steps include modeling these theory of minds and confidence levels in this multi-participant human computer interaction setting.

6 Resources

Website: https://www.baihan.nyc/viz/SupervisorBot

Video: https://youtu.be/X3bFx5bF95s

Codes: https://github.com/doerlbh/PsychiatryNLP