PROgress Implementation Toolkit

This website was made possible by funding from ViiV Healthcare, a global specialist HIV company established in 2009, majority owned by GSK, with Pfizer and Shionogi as shareholders

Authors

Rob J. Fredericksen, PhD, MPH – University of Washington, Medicine, Seattle, USA
Duncan Short, PhD – ViiV Healthcare, Global Implementation Science, UK
Emma Fitzsimmons, BA – University of Washington, Medicine, Seattle, USA
Justin McReynolds, MS – University of Washington, Health Informatics, Seattle, USA
Sierramatice Karras, BS – University of Washington, Health Informatics, Seattle, USA
William Lober, MD – University of Washington, Health Informatics, Seattle, USA
Heidi M. Crane, MD, MPH – University of Washington, Medicine, Seattle, USA

Foreword

This PRO Implementation Toolkit was assembled by the PROgress Study Team from the University of Washington and ViiV Healthcare, and reviewed by the PROgress Study Steering Group. The purpose of this document is to provide practical insights gathered from the implementation of patient-reported measures and outcomes (PROs) into routine HIV care to support those who may be considering this process.

PROgress is a research workstream entitled ‘Improving HIV care through the implementation of PROs within routine patient management’. This is comprised of three complementary components:

1. An implementation science research study (the PROgress study), integrating PROs into two HIV clinical care settings. This served two purposes: first, to identify the essential program elements that can support the sustainable implementation of PROs into routine HIV care in community settings; second, to examine the added value of implementing PROs into routine HIV care for the salient stakeholders, including the patient, the providers, and other clinic staff.
2. An Evidence Review and Summary, which is designed to raise awareness of the evidence relating to PROs in the care of people living with HIV (PLHIV) and to outline the potential for well implemented instruments within routine care. It draws on evidence from published literature characterizing the impact of PROs in routine clinical care for patients with chronic comorbidities including HIV-related literature as well as other fields, particularly oncology
3. This PROgress Implementation Toolkit, which is a resource for those considering implementing PROs in clinical HIV care. This is designed to provide practical advice to support the introduction of clinical PRO assessments into routine HIV care. These insights draw from a range of sources, including: practical experience integrating PROs into HIV clinical care at multiple sites, including the PROgress sites; published literature; and additional primary interviews with stakeholders that have experience integrating PROs into HIV clinical care. This Toolkit was designed to provide resources, tips, and learning to help implement PROs adapted as needed for individual clinics.

The PROgress Implementation Toolkit and Evidence Review and Summary, serve as complimentary resources, with the Toolkit designed to provide practical hands-on approaches and the Evidence Summary designed to summarize available real-world evidence supporting the integration of PROs within routine HIV clinical care settings with each informing the other.

Introduction

What is a PRO?

A patient-reported measure or outcome (PRO, also less commonly known as PROM) is defined as “any report on the status of a patient’s health condition that comes directly from the patient without interpretation of the patient’s response by a clinician or anyone else”.¹

PROs provide the patient perspective of the effects of disease and treatment including comprehensive assessment of factors such as mental health symptoms like depression/anxiety, well-being and satisfaction, health behaviors such as medication adherence, risk behaviors such as substance use and sexual risk behavior, as well as other social determinants of health and practical or safety information such as housing status and intimate partner violence.

How PROs integrate into clinic flow

What might the end result of PRO integration into care look like? How will it fit into existing clinic flow? Figure 1 below, based on clinical integration of PROs in the PROgress study in two HIV care clinics, provides an overview of what happens and when.

Figure 1. PRO implementation example in HIV clinical care

Evolution of PROs in care

While historically PROs have had a much larger role in research than care, the use of PROs in clinical care has been increasing as a result of several key developments. These include the rapid progression of technological infrastructure leading to the expanded incorporation of touch-screen tablets, internet-based applications, and electronic health records (EHRs) in clinical care.2 Furthermore, PROs are increasingly demanded by regulators, payers, accreditors, professional organizations, and providers to measure and address PROs at the level of the patient, clinic, and healthcare system as well as provide population information.² Legislative demands to improve healthcare outcomes without increasing costs has put more emphasis on quality of care, value-based reimbursement, and patient engagement. As such, PROs have been increasingly identified as the most direct and relevant measure to demonstrate high-quality patient-centered care.² The most common reasons cited for implementing PROs in clinical care among stakeholders who have already implemented them included screening, monitoring, treatment evaluation and treatment planning, and quality improvement including that mandated by external agencies.³ Other reasons cited in interviews of respondents from a range of healthcare settings in which PROs had been implemented included shared-decision making between patients and providers, and, less often, reasons related to satisfaction or reimbursement.³

Relevance of PROs to modern HIV care

Advances in antiretroviral therapy (ART) over the past decades have increased the life expectancy of PLHIV and transformed HIV from a fatal disease to a chronic manageable condition.⁴ The associated decline in mortality since ART has been introduced has led to increased emphasis on managing quality of life (QoL) and comorbidities, including those associated with HIV and its treatment. Many of the symptoms, health behaviors, and life circumstances associated with living with HIV and these comorbidities are not directly observable and are more easily measured by direct patient report. Yet, many such variables are under-addressed and not measured well in clinical care: in HIV care, examples include antiretroviral (ARV) medication adherence, substance use, sexual risk behavior, and depression.⁵ Reasons for this have included social desirability bias, time constraints, limited communication skills to convey symptoms or feelings, or linguistic and/or cultural barriers.^6-9 PROs help address these barriers. On-site PRO collection prior to routine clinical care appointments, via hand-held computer tablets with real-time results available to providers during clinic visits, has improved provider ability to detect and address depression/suicidal ideation, inadequate ART adherence, and substance use in HIV care.^5,10 Integrating PROs into clinical care of patients with chronic conditions, such as cancer, rheumatoid arthritis, and HIV, have been shown to be acceptable to patients and providers and valuable in clinical care;^10-12 they have improved patient-provider communication^13-17 and increased patient satisfaction with care.^16,18-20

Evidence-based support for PROs in HIV Care

PROs have been highly useful to providers and acceptable to patients. An in-depth review of evidence (PROgress Evidence Review and Summary) details this here and supports the idea that implementing PROs in HIV care can:

1. Improve detection of health behaviors, symptoms, and mental health issues
2. Increase both provider awareness and interventions to address depression, drug and alcohol use, intimate partner violence, and other domains in order to improve health outcomes
3. Improve patient-provider communication, by helping patients prioritize and raise concerns, and by helping providers identify and initiate discussion of less-observable issues and/or discussion of topics that are highly sensitive or personal to the patient (e.g. depression, substance use, sexual risk behavior)
4. Improve delivery of care, for example, allow providers to focus on the most relevant issues during the visit, increase referrals, more closely monitor treatment, and improve symptom management
5. Improve outcomes such as depression scores and symptom burden.

Toolkit purpose

This Toolkit provides practical advice to support the introduction of clinical PRO assessments into routine HIV care. These insights draw from a range of sources, including practical experience integrating PROs into HIV clinical care at multiple sites, published literature, and interviews with stakeholders with experience integrating PROs into HIV clinical care. While some of the information in this Toolkit applies to all formats of PRO assessment, we focus on implementation of tablet-based patient self-administered PRO assessments in clinical care due to the clear advantages outlined in Chapter 3. Given differences between HIV clinics, a ‘one-size-fits-all’ approach to PRO integration is unlikely to fit every clinic’s needs. With this in mind, this Toolkit provides resources, tips, and effective practices to help implement PROs adapted as needed for individual clinics.

Chapter overview: steps toward implementation

Though the process of PRO implementation is iterative, the chapters of this Toolkit sequence tasks into a general chronologic order starting with planning and decision-making (Chapters 1–3), then implementation (Chapters 4–6), and finally ongoing collection, maintenance, evaluation, and improvement (Chapter 7).

Chapter 1 helps assess and improve a clinic’s readiness for PRO implementation.

Chapter 2 offers a pathway for stakeholder engagement.

Chapter 3 itemizes steps needed in order to build technical infrastructure for electronic data collection.

Chapter 4 operationalizes important steps to create a PRO assessment that best suits the needs of an individual HIV clinic and its patients.

Chapter 5 outlines the decisions and protocols to support integration and ongoing success.

Chapter 6 offers insight into initial and ongoing staff training needs.

Chapter 7 provides strategies for monitoring, evaluating and sustaining the success of integration of PROs into clinical HIV care.

How this Toolkit was developed

Evidence and practical tips found in this Toolkit are drawn from real-world PRO implementation experiences and data collected from:

1. The PROgress study, a ViiV-funded project evaluating impact, effectiveness, and sustainability of PRO collection in routine clinical HIV care at Midway Specialty Care in Ft. Pierce, FL, and St. Michael’s Hospital in Toronto, ON. The study and Toolkit were developed in conjunction with a Steering Committee comprised of HIV care providers, PLHIV, and HIV care researchers
2. The Centers for AIDS Research (CFAR) Network of Integrated Clinical Systems (CNICS), a network of eight United States (US) HIV clinics. As of early 2020, >85,000 clinical PRO assessments had been completed by >20,000 PLHIV across CNICS sites as part of routine clinical care visits to improve care and facilitate research on health domains important to long-term outcomes among PLHIV.

Language

There are a number of acronyms throughout this Toolkit. They are defined in Appendix 1. We also define terms that may be unfamiliar or whose meaning differs by context with use. For example, we use the term provider to refer to the physician, fellow, nurse practitioner, physician assistant, or other clinician who is providing patient care. We realize in different contexts, terms such as provider have broader or narrower meaning.

Resources

• PROgressHIVcare.org

1. ASSESSING AND IMPROVING READINESS TO IMPLEMENT PROs IN HIV CLINICAL CARE

1.1 Are PROs right for my clinic right now? If not, how do we get there?

The appropriateness and readiness of PRO integration in your clinic depends on many factors including the patient population, needs and perceptions of clinic leadership and providers, logistics, technical capacity, and cost. The tool below is designed to help you assess and improve the feasibility of integrating PROs in your clinic within each of these dimensions, by helping envision how to overcome barriers.

Patient population:

Does the majority of the clinic patient population possess literacy skills to read at 6th grade level (approximately ages 10–11)? Yes/No

If ‘No’ (or unsure): PROs that are based on text only may be appropriate for only a subset of the population. An enhancement to text is use of a pre-recorded voice to help guide patients through a brief set of PROs. While this has advantages if reading levels are low, it adds a great deal of time/patient burden to the assessment and therefore should be considered only if needed or only for those patients who need it.

Does the majority of the clinic patient population possess the cognitive and physical capacity to complete a brief PRO assessment? Yes/No

If ‘No’: self-administered PROs may be less appropriate for these patients, and time may be better spent eliciting verbal self-report or the report of caretakers. Clinic staff may need a consistent means of distinguishing these patients from those with the ability to self-administer PROs. Of note, the ability to complete a brief PRO assessment on a tablet has been found to be feasible in many patient populations including the elderly (particularly if no mouse or keyboard as in tablets),^21,22 and broadly among diverse populations of PLHIV.^23,24 This is often much more feasible than expected.

Clinic leadership:

Is the clinic’s leadership likely to support the implementation of PROs? Yes/No

If ‘No’: consider what is driving this perceived lack of support. What are the clinic leadership’s key goals and priorities, and what evidence regarding the use of PROs may align with them? Leadership typically supports PRO implementation for a variety of reasons including improving patient care, better assessing needs, and enabling better data collection for administrative tasks such as mandated reporting and quality assurance from external agencies (See Chapter 2 – Engage Stakeholders).

Is there an individual or individuals on staff that can cultivate stakeholder interest in PRO collection and champion PROs as a priority? Stakeholder support or clinic champions have been critical for successful integration of PROs in clinical care.⁶ Yes/No

If ‘No’: see Chapter 2 – Engage Stakeholders which offers support for illustrating the benefits of PROs in clinical care.

Is there an individual or individuals on staff that can champion PRO data collection with respect to managing day-to-day operations? Yes/No

If ‘No’: consider what steps would be needed to identify such an individual or allot time in an existing individual’s duties that would be accountable for the ongoing success of this operation. For example, this individual may supervise front desk staff, or may be a designated medical assistant (MA).

Providers:

Are clinic providers supportive of the use of PRO measures in clinical care? Yes/No

If ‘No’: consider the basis for this. What are their perceptions of the value of PROs? What experiences form their basis for the lack of support? What are their concerns (see Chapter 2 – Address Common Concerns)? Have your providers had negative experiences with PROs? To what extent could these concerns be addressed? Provider concerns often focus on potential impact on clinic flow or visit length. Does focusing on specific, difficult-to-assess domains relevant to improving care, such as substance use or intimate partner violence, impact support? Does focusing on a brief assessment, with plans to implement in such a way to minimize impact on flow, minimize these concerns?

Has their response to changes in prior clinic protocols been positive? Yes/No

If ‘No’: identify the key factors driving this. Evidence supporting the benefits of PROs to improve clinical care may minimize these concerns (see Chapter 2).

Logistics and flow:

Can the clinic allot time at the beginning of the visit for patients to take a PRO assessment without disrupting flow? Yes/No. Note: amount of time needed will depend on assessment length.

If ‘No’: do opportunities exist to collect PROs from patients while they are waiting for their provider, or off-site, using personal electronic devices?
Note: while off-site PRO administration may lessen the impact on clinic flow, as a standalone approach it excludes patients who lack such devices. It may however be useful as a supplementary approach to within-clinic PRO collection to decrease impact on clinic flow. This may be a particularly useful approach in care settings with telehealth visits, an increasingly common practice as a response to the COVID-19 pandemic. See Chapter 5 – Outline Workflow. A PRO timing estimation tool is available here.

Does the clinic have a plan to allot space on-site for patients to self-administer a PRO assessment? Yes/No

If ‘No’: can patients complete the PROs while they are waiting for their provider in the examination rooms, or off-site? See above note regarding off-site PRO administration.

Technical capacity:

Does the clinic have the technical capacity to support electronic data collection? Yes/No

If ‘No’ (or unsure): see Chapter 3 –Technical Choices and Infrastructure.

Cost:

Can the clinic afford the resources necessary, such as the staff time and equipment (e.g. iPads or other tablets)? Yes/No

If ‘No’ (or unsure): see the next section Start-up and recurring costs.

1.2. Startup and recurring costs

Understanding the financial costs associated with implementing PROs is fundamental to long-term success. There are different types of costs to consider: initial start-up (or one-time capital costs) and recurring fixed costs.

Start-up costs bring a project to operational status (e.g. software development, purchase of office equipment, licenses, etc). These costs are incurred at the beginning of the project or at a single point in time, and not as a year-to-year or month-to-month expense.

Table 1 contains typical start-up costs to consider when developing a program budget.

Of note, some costs (e.g. internet) will not apply to many clinical settings, as these resources are already available. Personnel are likely the most important component of costs in many clinics, whether it is adding new staff or reallocating duties. IT costs can include platform development versus integration of existing platforms (several are available as shareware but will still require programming [see Chapter 3]).

Recurring costs occur on a regular basis, and typically fall within an annual budget period. Unlike one-time costs, recurring costs generally remain the same within the budget year (e.g. general administrative costs, rent, license renewal, etc). However, normal price increases (e.g. rent increases, pay raises, or other cost-of-living increases) should be budgeted for each coming year. Again, many of these will not apply to many clinic settings however, personnel considerations are key.

Table 2 shows examples of recurring costs.

Again, some costs, such as internet use, will not apply in most clinical care settings (e.g. if internet resources already exist), and cost of personnel time is likely the most significant.

Sample cost itemization tool

Below is an itemization of start-up and maintenance costs for integrating and administering PROs. Enter costs in the boxes below to determine:

Fixed Costs

Device or devices (e.g. iPads or other tablets). You may wish to purchase more than one device in order to administer PROs to more than one patient at a time:

Printer, if PRO results are printed out (printer is not a cost if integrated into EHRs or presented to provider on-screen at start of visit):

Recurring costs in 1 year

WiFi access (monthly fee X 12 months):

Paper, 1 page of PRO results per patient (no cost if electronic) multiplied by number of visits for which PROs are administered in a year:

Toner for printer. Number of cartridges needed depends on number of PROs administered. A standard cartridge prints 220 pages:

Salary associated with % full-time equivalent (FTE) for staff member setting up and overseeing use of tablets. In the PROgress study, staff estimated a maximum of 4 minutes per patient for explanation of procedure, setup, collection of device, and delivery of paper-based results. This estimate is based on the one-on-one interaction time of the staff member with the patient as well as time handing the tablet, and not the full completion time of the PRO assessment by the patient as the staff member often left the room and did other activities while the patient completed the assessment:

Total :
Sum

One of the biggest issues that lead to delays and practical barriers including unplanned costs with PRO implementation, involves integration with the EHR. Creating new platforms can be expensive and time consuming, integrating with EHRs is sometimes not feasible and the burden of the slow and/or complex EHR control processes have been noted as a hurdle.3 This is even before taking into account the frequent changes in EHR products that are occurring across many healthcare settings. Fortunately, stand-alone platforms can be used that are much less expensive than developing new platforms, that allow PRO collection to continue regardless of EHR changes and facilitate implementation with PRO feedback even before EHR integration.

PRACTICAL TIP
A plan regarding technical approaches (Chapter 3) is important when estimating costs. EHR-based approaches are not necessarily the most effective (due to many PLHIV not being linked to patient portals), efficient, modifiable, or practical. However, this is a quickly moving area and will need to be assessed on a site-by-site basis (Chapter 3).

1.3. Creating a business case for PRO implementation

Your organization may require a formal business case for PRO implementation in order to identify short- and long-term goals and associated budget requirements. This may include needs, solutions, approaches, risk assessments, and value analyses.

Table 3 shows an example of this format.

2. ENGAGE STAKEHOLDERS

2.1 Identify stakeholders

Early stakeholder engagement facilitates successful PRO implementation and its sustained use. Stakeholders include clinic leadership, patients, providers, and staff, and may also include others, such as hospital administrators or researchers. Each brings a unique perspective, concerns, and valuable input.

The broad goal of improving care tends to engage stakeholder interest, with very specific concrete examples such as identifying otherwise undetectable suicidal ideation, depression, or inadequate adherence to ART, being very relatable goals that almost all stakeholders can appreciate. Furthermore, these types of goals have a strong evidence base supporting their likely success.

Individual stakeholder goals may vary. Early preliminary discussions to understand stakeholder concerns and goals can help tailor information and secure valuable support. For example, providers may be more motivated to adopt PROs to increase the appropriate diagnoses or identification of issues such as inadequate adherence, nurse managers may be concerned with impact on flow, patients may wish PROs included areas such as HIV stigma or social support to provide context to health needs, and administrators may want to know if PROs can help satisfy external reporting requirements such as the percentage of clinic patients who complete a mental health instrument. Regardless of the stakeholder or motivation, engagement early in the process can identify and address benefits and concerns around PRO implementation (see Address Common Concerns in this chapter).

2.2 Prepare demonstration of value

Stakeholder interest depends on clear demonstration of value relative to cost. Ideally, a value demonstration takes the form of a brief presentation or a simple summary sheet. A presentation demonstrating the impact of PROs on provider awareness and documentation of depression, substance use, and inadequate medication adherence is here.⁵ While clinical benefits such as increased provider awareness of risk behaviors are relevant for all HIV clinics, additional benefits may be applicable for specific settings (e.g. reporting requirements or meeting annual depression screening requirements for specific state-based payees). Providers have particularly valued improved identification of suicidal ideation and substance use.^10,25

In the case of PRO adoption in the clinical setting, a value demonstration should consider who the beneficiaries are (e.g. the patient, the provider) and what the advantage unique to the stakeholder might be (e.g. better quality patient data for diagnosis, improved provider/patient interaction, more comprehensive care, and improved health outcomes for the patient).

PRACTICAL TIP
Provide a clear justification for PRO data collection, as providers and staff are more likely to support PRO implementation if they understand the value.³ Short practical examples are more compelling.

Example 1: Improve patient engagement in care via better detection of treatable problems. On average, ~30% of PLHIV in the US report depression at any one time, and this is associated with many poor outcomes such as mortality. 26 It is notable that even among PLHIV with known depression, there are substantial gaps in the depression treatment cascade with lack of follow-up to see if treatments are effective. ²⁷ We plan to integrate a brief clinical assessment of PROs including depression to identify the ~20% of our clinic with undiagnosed or undertreated depression to improve care for those PLHIV.

Example 2: Reduce preventable death. Two patients in the past year were killed as a result of intimate partner violence (IPV). The Centers for Disease Control and Prevention (CDC) has estimated that the rate of IPV among women with HIV is double the rate of those without HIV (~55%). 28 PROs may effectively detect IPV. We are also seeing increasing heroin overdoses. We would like to implement a brief clinical assessment of PROs to allow a standardized approach to screening for IPV, drug and alcohol use, depression, and inadequate adherence, to allow us to better identify those PLHIV in our clinic who may benefit from additional interventions such as addressing IPV and ensuring Narcan for those using illicit opioids.

2.3. Meet with stakeholders

Meeting with diverse stakeholders as a group presents an opportunity to describe the purpose of PROs, alleviate concerns or misapprehensions, and to coalesce support for their known value in improving care and clinical outcomes.5 An adaptable sample presentation for this purpose is here.

Beyond the presentation of static information, we recommend giving stakeholders the opportunity in this meeting to interact with PRO measures and their output in order to better understand the process and its potential. This step, in our experience, has been a key turning point in generating enthusiasm and helping stakeholders visualize integration into daily routine. An example of an electronically-administered touch-screen PRO assessment for this purpose is here, which your audience may self-administer on their own or other devices. Sample output, or the results generated for providers is here and in Figure 6 (Chapter 4). An interactive tool for determining average timing to completion for varied combinations of individual PRO measures is here. Screenshots of the interactive tool can also be found in Appendix 2.

Figure 2 shows a sample agenda for an initial meeting with stakeholders.

A sample presentation for this purpose is in Appendix 3.

PRACTICAL TIP
Bring tablets and a demonstration version. Showing how straightforward it is for PLHIV to complete and get started with an assessment is crucial to alleviate staff concerns. Shorten the agenda as needed to ensure adequate time for the demonstration.

2.4. Provide overview of value

A comprehensive overview of the potential value of PROs is available in the form of the PROgress Evidence Review and Summary, as a companion document to this Toolkit.

2.5. Address common concerns

Stakeholders will likely have many valid concerns about introducing PROs into their practice. Below is a list of common initial stakeholder concerns regarding the implementation and ongoing use of PROs. Many of these concerns have proven addressable in past integration efforts.

Table 4 summarizes common concerns, followed by suggestions regarding how they can be effectively addressed.

2.6. Include providers in PRO selection process and output design

Without clear relevance or usefulness to providers, PROs will be less likely to succeed long-term in the clinic. Therefore, consider early engagement of providers to determine what PRO domains would be of most use to improve their ability to provide the best possible care for PLHIV, and by extension, patient outcomes.

Providers are the end-users of PRO data. The results need to be highly relevant, interpretable, and easy to interact with. To help ensure this, always include providers in decision-making processes on the content, format, and design of PRO results. This includes what form of presentation the reporting will take (e.g. paper vs electronic, multiple vs single time points), how scoring is displayed and explained (e.g. Patient Health Questionnaire – 9 items for depression [PHQ-9]), organization of information (e.g. which domains at the top of the page vs bottom), and aesthetics (e.g. use of color or bolding to highlight pertinent information). Early provider engagement helps ensure inclusion of clinically-relevant, actionable domains, and ultimately, investment in the PROs as a clinically-useful tool tailored toward your clinic’s specific needs. See the sections titled Select PROs: domains and attributes to consider, Format results and Create PRO assessment in Chapter 4.

PRACTICAL TIP
A very common concern among providers is that PROs will lengthen visits. ³ When presenting plans to implement PROs to stakeholders, emphasize evidence showing this is not likely the outcome. ²⁹ Demonstrate the actual time to complete the assessment during stakeholder discussions using tablets and a demonstration assessment. Propose a plan for minimizing impact on flow, such as having

PRACTICAL TIP
Impact of PROs on clinical flow is an almost universal stakeholder concern. ³ Reassuring stakeholders with a clear plan to start slowly in order to minimize impact on care processes is crucial. PROs can then be expanded once any implementation issues have been addressed. Examples of successful roll-out approaches that have been used to minimize impact on flow include starting on the least-busy clinic days of the week, starting with one full-time provider invested in the success of PROs and able to work with the team during integration, and starting with a very targeted brief PRO assessment.

2.7. Secure implementation champion

One effective strategy for building momentum around PRO implementation is to identify or cultivate a ‘champion’ in the setting where PRO implementation will take place. This person is tasked with advocating for the use of PROs, inspiring the range of stakeholders and increasing/maintaining commitment to maintain momentum of PRO adoption. The champion will remain close to the process as it evolves and will serve as a communication conduit for all parties, including eliciting arising concerns and issues, and communicating plans and successes in navigating these as the project evolves. This person is often the clinic director – with a PRO coordinator acting in this role once implementation occurs. Champions provide leadership, guidance, and encouragement to stakeholders, and focus on the sustainability of the PRO implementation.

3. TECHNICAL CHOICES AND INFRASTRUCTURE

3.1. Understanding PRO Choices

A successful PRO implementation should integrate both treatment- and patient-centered perspectives into one health information system, which should be crafted to optimize the patient experience, to enhance the provider’s clinical use of the data, to minimize any challenges to clinic flow and efficiency, and to maximize population health utility of the information. In this context, population health may include clinic-level, research, and public health uses of the data.

The first major step is confirming that all stakeholders share the intent to implement computerized collection of PRO data. While PROs may be collected by paper, the manual approach, this will likely be dismissed once a variety of factors are considered. The disadvantages of paper-based collection include increased clinic workload to administer PROs and enter responses, decreased accuracy from required data entry, provider effort in summarizing data to realize clinical utility, and patient perceptions of both the utility and process of providing PRO responses. See Chapter 4 for more details regarding relative advantages and disadvantages.

The second step is deciding which of four types of PRO systems to consider:

1. a commercial “standalone” PRO system that can be integrated with an EHR
2. an EHR vendor’s “built-in” questionnaire tools
3. a developed PRO system developed “in-house”
4. a PRO system supplied as part of participation in a research or clinical network.

All EHR and standalone PRO system vendors highlight their system’s ability to gather PRO questionnaire responses, provide those responses to providers, and ensure they become part of the medical record. However, it is important to examine each of these four options carefully to compare: the true cost of acquiring, integrating and supporting any particular product; the experience of the patient in accessing the system and recording their responses; the experience of the provider in accessing and using a clinically-relevant summary of longitudinal patient data; and the ‘cost’ (licensing, implementation, maintenance) of persisting the data in the EHR – and if the data are persisted, whether that is done as a scanned image, an electronically-transmitted summary form and/or discrete observations. How the data are stored impacts the ability to derive secondary value, beyond clinical utility, from population health applications.

Often IT organizations strongly favor using tools from their existing EHR vendor, in order to avoid a new vendor relationship and contract, simplify implementation of interfaces, avoid a new set of security considerations, to advance an agenda such as increasing patient use of a patient portal system, or simply to make use of existing vendor support mechanisms. However, the impact of a choice made only on these technical and operational considerations can significantly impact the experience of the patient and the value to the provider, which should be paramount to maximize data collection and use.

Once the four types of PRO systems have been considered and an evaluation is determined, the approach that best balances function and cost, has provider support, and is acceptable to any applicable IT governance process, then the particular pathway choice will drive other technical needs (i.e. for software/operating system support, hardware platform, data storage attributes to support high data integrity, data center needs to ensure secure operation).

3.2. Identify Issues to Guide Choices

It may be helpful to organize the issues to be addressed according to the phases of gathering, using, storing, and reusing PRO data. These include:

1. The flow of information from the patient to the provider
2. The use of the information by the provider
3. Storing the information to meet medico-legal requirements for medical record collection, retention, and accounting of disclosures
4. Reuse of the data for population health purposes.

Gathering information from the patient:

• In what settings will the patient use the system? Only in the clinic? At home, or in another location of their choosing?
  - What kind of support can be provided for in-clinic use?
  - In what locations can the patient use the system prior to their clinic visits?
  - What support is available for out-of-clinic use?
  - Can in-clinic use serve as a “backup” plan for patient’s unable to complete PROs outside the clinic, and if so, how does the communication occur to ensure that the patient can be prompted?
• Can the system deliver reminders to the patient to complete PROs, or to the staff that a PRO has not been completed prior to a visit?
  - What hardware will deliver those reminders and what constraints does that place?
• What devices will the patient use in clinic? Tablets or iPads are a common choice.
  - Can concerns about damage or loss of in-clinic devices be assuaged?
  - Are there valid concerns about device cleanliness?
  - How will defective or mis-configured devices be identified and repaired?
  - What level of duplicate devices are needed to ensure tolerance of faulty hardware?
• Can the patient use the device of their own choosing out of the clinic? Can they use that device in clinic?
  - Can they use their phone?
  - Can they use a computer?
  - What security considerations are there if they are using a device that is not theirs?
• How do we validate the identity of the patient when initially enrolling them?
• How do we authenticate patients when they are using the system?
  - Is that done in person, for in-clinic use?
  - Do we sign them in with a patient-specific username/password and what kind of registration process do we use to support that?
  - How do we manage support for recovering passwords, or other login issues?
  - Does logging in present too much of a barrier to use and are there strategies such as one-time questionnaire links which can be used?
     o What are the institutional concerns around using one-time links?
     o How does it impact the data that can be shown to the patient?
     o How does it impact the value to the patient?
• What controls the patient’s periodic usage of the system?
  - Can the patient use the system electively, like a diary in their control, or only on a fixed schedule, as if it were an in-clinic questionnaire?
  - Can they edit/correct/revise their responses? How do they address errors?
  - Can the system deliver reminders to the patient to complete PROs, or to the staff that a PRO has not been completed prior to a visit?
  - What hardware will deliver those reminders and what constraints does that place?
• Do we need to implement a patient’s “right be forgotten” in a particular setting?
  - Does that conflict with the need to maintain records, or to make data available for population health uses?

Presenting information to the provider:

• How can the information be presented to the provider?
  - Discrete values sent to an EHR and viewed through charting or table functions?
  - Customized, domain-specific longitudinal patient summary?
• How can the provider access those views of information?
  - While logged into the EHR, using EHR tools?
  - While logged into the EHR, viewing a static summary?
  - While logged into the EHR, using an application within the EHR to view the data interactively?
  - Similarly, the last two options may be options in a vendor system, locally developed system, or system provided as part of a research or clinical care network.
  - Are there visualization considerations that create opportunities to overlook or erroneously interpret information?
  - Especially if patients can use the system electively, are there visualizations which efficiently present a large amount of information?
• Is viewing information closely linked to the providers documentation process?
  - Are there “macros” or summary information that can be imported into the documentation?

Storing information into the medical record:

• Are the PRO data preserved as discrete observations in the EHR?
• Are they preserved in their entirety, or only in the form of scores or alerts?
• Are they preserved as a visit-specific longitudinal summary, imported or preserved in the EHR system and linked to a specific encounter?
• If not part of the EHR, are the data preserved in a system that meets reliability, security, assurance, and retention requirements under appropriate regulations?
• Are there other more general regulatory or policy considerations, such as privacy practices or security regulations or Institutional Review Board (IRB) guidance, with which the system has to comply?

Reuse of information for population health:

• Are there individual level metrics of frequency and completeness of use? Is individual level data collected from patients and providers to monitor satisfaction and barriers to use?
• Are there clinic level indicators of use available to show broad patterns of use and monitored for changes that might represent systematic barriers or enablers to use?
• Are there clinical level metrics based on aggregate outcomes of clinical significance, so that PRO data may be used to monitor patient-reported impacts of changes in care patterns?
• Are adequate data being collected to satisfy any research or public health goals for secondary use of those data?
• Are the PRO data represented using standard formats and value sets, such that extracted data may have its structure and meaning consistent with that form other systems?

As an example, in the PROgress study, after discussion with teams at St. Michaels Hospital and Midway Specialty clinics, including discussions with the IT staff, we addressed these issues as follows:

Gathering information from the patient: Initial implementations at both sites were based on in-clinic administration of the PROs on a tablet. This simplifies both the direct support for the patient in using the system, if needed, as well as the issues of identity verification and account management for the patients. However, this comes at a cost of flexibility of administration. Both sites were pleased with their choices, but both had trouble making the transition to remote use with the onset of COVID-19 concerns and the shift to telehealth visits.

Presenting information to the provider: Both sites valued integration with the EHR, however both were in transition to, or strongly considering new EHR vendors and would have needed a clear and economical path to integration with their existing vendors. We did explore both Fast Healthcare Interoperability Resources (FHIR) interfaces and Substitutable Medical Applications, Reusable Technologies (SMART) on FHIR integration with one of the vendors, but exchanging enough information to secure a firm bid proved elusive and both sites elected to use automated printing of a results summary on PRO completion, and delivery of that summary to the provider at the time of the visit.

Storing information into the medical record: Both sites elected to scan the summary sheet into the electronic medical record since an electronic interface was not implemented and they were soon going to transition to other EHRs.

Reuse of information for population health: Both sites elected to rely on the built-in dashboard features in the administrative view to track usage and completion, and on the analysis data set download feature for further examination of population level responses. Midway requested a ‘whiteboard’ view to track waiting room completion status in real time in the provider room, which was implemented. This is similar to a feature that had been previously developed for another clinical setting and allowed a screen to always show progress of patients on the PROs in the clinic in real time.

3.3 Consider System Features

Table 5 describes considerations for the PRO system features, adapted from Fenton et al,₃₄ which reflect a similar set of concerns.

3.4. Consider Data Quality

System features may include creating a dashboard to enhance usage monitoring, improve data accessibility, and monitor data quality. Minimizing data errors within the PRO system is critically important. Errors or missing data can be reduced through automated data quality assurance measures that assess data for inconsistencies (e.g. validation rules built into the application), and through consideration of issues that discourage patients from starting or completing PRO sessions.

Figure 3 describes the characteristics that define data quality.

Figure 3. The characteristics that define data quality

Adapted from Fenton et al.³⁵

Accuracy and precision refer to the exactness of the data. Accuracy in healthcare is worth high levels of investment. The data must not contain errors and must convey the correct information without being misleading. PRO answers that are considered valid or legitimate based on the survey’s requirement are allowable.

At the same time, it is important to realize that requiring a patient to answer a question does not guarantee accurate data, any more than requiring a healthcare provider to acknowledge an alert guarantees their thoughtful consideration of the underlying information. Patients should be given a pathway through the PRO process which encourages them to provide information useful to their care, and to understand the benefit of providing that information.

There must be a valid reason to collect the data to justify the time and effort required. PRO data collected that is not relevant can misrepresent a patient’s health status and drive inaccurate clinical decision-making. Incomplete data collection can lead to incomplete understanding of patient health. Providers need the right level of access to the PRO data to adequately evaluate the data in a timely manner.

There must be a reliable mechanism that collects and stores the PRO data without inconsistency or variance. The level of data detail is important, since inaccurate decisions can occur if the data is not clearly presented. Simple raw data may have a different meaning than data that has been aggregated and summarized.

A PRO system that offers a wide range of software features can adapt to specific patient or provider needs without significant additional programming and supports reporting and data visualization solutions.

3.5. Resources

Guidance on Infrastructure

• Snyder C and Wu AW, eds. Users’ guide to integrating patient-reported outcomes in electronic health records. Baltimore, MD: Johns Hopkins University. 2017. Funded by Patient-Centered Outcomes Research Institute (PCORI); JHU Contract No. 10.01.14 TO2 08.01.15. Available here. Accessed October 2020.
• Coons SJ, Eremenco S, Lundy JJ, et al. Capturing patient-reported outcome (PRO) data electronically: the past, present, and promise of ePRO measurement in clinical trials. The Patient 2015;8:301-9.

Other Helpful Information or Examples

• ePROs in clinical care: guidelines and tools for health systems. 2020. Available from: https://epros.becertain.org. Accessed October 2020.
• Clinical data capture and management evaluation checklist. Available from: Oracle Data Sheet.
• Review of data accessibility methods in healthcare. Available from: https://www.researchgate.net/publication/280722426_REVIEW_OF_DATA_ACCESSIBILITY_METHODS_IN_HEALTHCARE. Accessed October 2020.
• The MAPS Toolkit: mHealth Assessment and Planning for Scale. Geneva: World Health Organization; 2015. Available from: https://apps.who.int/iris/handle/10665/185238. Accessed October 2020.
• REDCap (Research Electronic Data Capture) is a secure web application for building and managing online surveys and databases. Available from: http://projectredcap.org. Accessed October 2020.

4. CREATE PRO ASSESSMENT

There are several decisions to consider while creating the PRO assessment. These include determining the mode and frequency of administration, which PRO measures to include and in what order, what skip patterns are needed within the assessment, and how the results will look. This section helps fully consider each of these decisions.

“What do we want to know about our patients that would allow us to do a better job — to increase our knowledge and take action?”
- Clinic Director, speaking about their implementation experience and PRO selection.

4.1. Decide mode of administration

PRO measures may be administered in electronic or paper format. We briefly describe the advantages of each below.

Advantages of paper administration:

• May be more familiar to patients with low computer literacy
• Quick, lower-cost start-up.

Advantages of tablet-based administration:

• Ability to automate skip patterns within assessment so that patients only receive relevant questions (e.g. no smoking frequency questions shown if indicates not currently smoking), reducing time burden. The ability to integrate skip patterns dramatically reduced patient burden and therefore impact on clinic flow. It may be one of the most important advantages of tablet-based collection
• Patients prefer or perceive advantages to tablet-based over paper-based administration^21,22,36
• Automated scoring within domains (e.g. PHQ-9). This reduces errors and also staff burden as no scoring by staff is required. It facilitates having scored results available to provider in real-time to enable clinical care impact
• Ability to link PRO responses to real-time pager alerts to clinic staff for high-risk patients, such as when suicidal ideation is endorsed. Among clinics that are also doing research, can use same approach to automate pages for other reasons including study recruitment
• Ability to administer in multiple languages yet easily interpret results
• Generation of real-time, comprehensive summary of results, with potential to illustrate differences between time points using graphics
• No physical space required for paper feedback form data storage.
Furthermore, depending on goals, additional benefits can include:
• Potential for remote administration such as for telehealth visits with real-time results delivered to clinic staff
• Programming flexibility allows for patient-specific administration, such as showing specific PROs to select patients at select time intervals based on historic responses or risk factors
• Ability to synchronize with audio accompaniment when needed for PLWH with lower literacy levels or those with poor vision
• Easier to compile population-level data for analysis
• Potential for information to populate EHR as discrete data.

While paper-based administration may be easy to implement quickly at a low start-up cost, we strongly recommend tablet-based administration given advantages in reducing patient burden through algorithmic and skip-patterned administration, automated real-time alerts and scoring interpretation, and staff data entry burden, as well as patient preference. Given the burden on staff of scoring instruments when using paper-based collection, using a computerized approach has also been found to be less expensive in the long run for doing anything but the smallest number of assessments^36,37 and tablet prices have recently been continuing to decrease.

In Figure 4, we offer considerations when selecting and programming a tablet-based mode of administration.

PRACTICAL TIP
Adding audio options to tablet-based PRO collection may be useful if working with a very low literacy population or those with vision problems, but for many patients it will increase completion times substantially and therefore should be avoided in order to minimize patient frustration and impact on flow.

4.2. Select PROs: attributes to consider

A key consideration when implementing PROs is how the information will support clinical decision-making and improve an individual patient’s care.

• What would be helpful to know about patients that cannot be easily or necessarily revealed through lab results or direct observation during the visit?
• What lines of inquiry might patients more comfortably and comprehensively answer in a computerized assessment, relative to face-to-face discussion?
• Most importantly, what PRO domains or health topics are directly actionable and eliciting and acting on the information would result in an improvement in care?

There are many domains of inquiry critical to HIV management and amenable to PRO assessments. Examples include depression/suicidal ideation, ARV medication adherence, alcohol/substance use, and HIV/sexual transmitted infection transmission risk behaviors. In addition, PROs offer an opportunity to explore social and context-based domains, such as partner violence, housing, social support, HIV stigma, and health-related quality of life (HRQoL). Figure 5 shows key factors needed for selecting PRO domains.

Once the desired PRO domains are identified, consider which PRO measure is most appropriate within each domain. To ensure consistent uptake and relevance, a PRO measure used in clinical care should ideally be:

• Brief – the smallest number of items possible to provide the desired insight
• Validated – accurately measures what it says it will
• Easy for patients to understand (comprehensible to 6th grade literacy level)
• Easy to recall (e.g. shorter vs longer recall periods, particularly for mundane behaviors)
• Interpretable by providers at a glance.

Note that while many PROs are free of charge in the public domain, others require licenses and sometimes require fees or developer notification before use. In addition, requirements for use may change over time. Many high-quality instruments are available that do not require a license for use and in most circumstances may be preferable.

Table 6 lists a number of PRO domains relevant to HIV care.

PRACTICAL TIP
Keeping assessment length short (<10 minutes) facilitates integration, minimizes patient burden, and decreases impact on clinic flow.

I think at first just start small. One of the things that we had to do too is prioritize our surveys… because you don’t want patients to sit there and complete 20 minutes’ worth of survey, (if) the providers are ready to see them. So, you have to be conscious of how much you’re asking the patient to do and what the impact will be like on their workflow.
- Implementer from a large US HIV clinic

4.3. Identify scoring and interpretation needs

In order for providers to easily review and process PRO results, choose measures with clear scoring and missing data guidelines. PRO results may have a range of different outputs of value. For example:

• Total scores based upon all questions
• Scores that are based on a discrete concept or sections within a PRO
• Scores based upon a single-item.

Some PROs require an algorithm that helps convert responses into scores that are easy to interpret and explain, such as a single number; some PROs simply require answer scores totaling up.

The scores can then be compared against an interpretation grid. For example, a depression score of 24 from a depression severity questionnaire may suggest ‘severe depression’; it is important to understand the sensitivity and specificity of the cut-offs.

It may also be valuable where a PRO measure provides:

• Reference scores for ‘similar’ patient groups that allow providers to compare their patients with similar patients e.g. those on the same type of medication
• Reference scores for PLHIV, which then allow the provider to compare individual patient scores with the average of a larger population
• Reference scores for general population, which allows the provider to compare an individual patient score with a normative score
• Linking scores to clinical practice guidelines.

The value in an approach that enables a comparison of the change in a patient’s score over time should also be considered.

An example of interpretation guidance is presented in Table 7 for the PHQ-9.³⁹ The PHQ-9 captures the frequency of depressive symptoms experienced in the past 2 weeks by asking 9 questions. The response options to each question are ‘not at all (0)’, ‘several days (1)’, ‘more than half the days (2)’, and ‘nearly every day (3)’. The PHQ-9 has been used to make a tentative diagnosis of depression in at-risk populations, and it has been validated for use in primary care.⁵⁸ To score the PHQ-9, the totals for each question are summed to reach a total score (maximum 27).

4.4. Determine order of PRO measures in assessment

The chronological order in which PRO measures are placed in the assessment may have an impact on your patients’ response. We recommend beginning the assessment with a relatively benign domain, before building up to more sensitive topics such as sexual risk behavior, intimate partner violence, or substance use. Another consideration is that you may want to place the measures with the highest clinical relevance (e.g. suicidal ideation, ART adherence) earlier in the assessment in order to ensure that the patient completes the most critical information.

4.5. Determine frequency of administration overall and for each measure

After selecting domains and measures of interest, consider ways to minimize the patient time burden for completing PROs. This includes length of time for completion of assessment, the frequency with which patients will be offered PROs, and the frequency with which patients will be shown specific measures.

A tool for estimating the average length of time your PRO assessment will take to complete can be foundhere.

To establish the desired frequency with which patients will be offered the PRO assessment, consider the intervals in which your clinic’s patients typically are seen for routine visits. You may wish to administer PROs no more frequently than every 3 months, every 6 months, or annually, depending on timing of visits and your patient population’s needs. More frequent assessment may yield richer information, however, administering PROs too frequently may frustrate patients and increase staff burden and clinic flow impact.

PRACTICAL TIP
Occasionally PLHIV come to clinic very frequently (e.g. for wound care, intravenous antibiotics, other reasons). It is therefore important to set a PRO eligibility window so that PLHIV who are in clinic multiple times in the same week are not asked to complete them repeatedly. It will both annoy PLHIV and impact clinical burden.

An electronic PRO platform can be programmed to show individual PRO measures at specific time intervals. Not all PROs may need to be administered at every visit. For example, gender identity and sexual orientation are less prone to change relative to other domains, and so might be asked only every 2 years or even less often. Other domains, such as family history of chronic conditions, only need to be asked once.

An electronic PRO platform can also be tailored to administer individual measures with varying frequencies, based on an individual patient’s previous responses. For example, for an older patient who previously indicated having never smoked in the PROs, it may not make sense to ask about smoking habits on each subsequent visit; this question could be skipped in lieu of domains more relevant to that patient, and revisited less frequently (e.g. every 2 years). Conversely, patients who in previous PROs endorsed specific symptoms or risk behaviors, might be shown corresponding PRO measures more frequently. For example, a patient that indicated intimate partner violence six months ago likely needs more frequent assessment than a patient that has consistently indicated not experiencing it.

4.6. Format results

Output generated from PROs must be easy to interpret at-a-glance in a busy clinic setting. Ideally the results should also be formatted in a way that supports sharing or communication to the patient.

Figure 6 shows a sample PRO summary report (based on a fictional patient) illustrating a means for displaying PRO results across multiple time points. In this example, time points are retrospective moving from left to right, with alerts indicated by bold exclamation points to get the provider’s attention. The use of color and shading may also promote readability.

Figure 6. Example patient-reported outcomes provider feedback

4.7. Build your own PRO assessment

After considering the PROs you would ideally like to administer to your patient population, as well as the frequency with which you would ideally administer individual measures, we recommend calculating the potential time burden in advance of implementation. Below is an interactive tool to help calculate the anticipated average time burden to completion for several commonly used PRO measures. Multiple selections calculate a total anticipated average time. This tool may help demonstrate to stakeholders the number and nature of PROs that could be potentially queried, and the anticipated time burden and impact on clinic flow.

try it yourself here

4.8. Resources

• International Society for Quality of Life Research (prepared by Chan E, Edwards T, Haywood K, Mikles S, Newton L). Companion Guide to Implementing Patient Reported Outcomes Assessment in Clinical Practice, Version: February 2018. Available from: https://www.isoqol.org/wp-content/uploads/2019/09/ISOQOL-Companion-Guide-FINAL.pdf. Accessed October 2020.

5. OUTLINE WORKFLOW

Once you have decided on the mode of administration, which domains of care to measure, which measures to use, the average length of the assessment, and the format of PRO reporting, you are ready to begin visualizing the specifics of integrating PROs into your clinic’s workflow.

5.1. Select workflow: when, where, and how to administer PROs

There are three key ways to integrate PROs into your clinic’s workflow. Each has advantages and disadvantages, described below.

Remotely, prior to appointment.

In this option, PLHIV are asked to complete a PRO assessment in their own time and on their own devices before the appointment begins. Results are either uploaded to an EHR or printed out before the appointment by clinic staff and delivered to the provider and relevant staff.

ADVANTAGES:

• Minimal impact on clinic flow
• Patient controls timing and location of response.

DISADVANTAGES:

• Difficult to respond to emergencies in real time, such as suicidal ideation or intimate partner violence (e.g., if endorsed after business hours)
• Excludes patients that do not have access to electronic devices.

On-site at the beginning of an appointment.

In this option, PLHIV complete the PRO assessment upon check-in for a clinic appointment. For many clinics, PLHIV wait long enough on average to complete a brief assessment before their provider appointment. Other clinics with limited wait times may need to schedule the appointment time 15 minutes earlier to allow enough time to complete the assessment. Front desk staff, or a MA sets up the assessment on a touch-screen tablet. This is often done in conjunction with measuring vital signs.

ADVANTAGES:

• Up-to-the minute information on patient health status
• Ability to respond quickly to emergencies, on-site and in real-time
• Creates agenda for how time may be best spent during appointment
• Includes everyone regardless of whether they have devices.

DISADVANTAGES:

• PRO completion is dependent on patient punctuality or provider delay, in order to avoid disruption of clinic flow.

A combination of both approaches.

In this option, PLHIV are asked to complete a PRO assessment in their own time and on their own devices before the appointment. Some PLHIV do not have access or will not complete the PROs remotely. Those who do not complete the PROs remotely are then asked to do it when they come to clinic.

Advantages include all of the above advantages. In addition, by having some of the PROs completed in advance, it decreases clinic flow burden. Finally, it provides options in the current era where a larger proportion of clinic patients are being seen via telehealth in response to the COVID-19 pandemic.

5.2. Define staff roles and centralize responsibility

Consider each staff member in the clinic and what role each of them play (e.g. provider, front desk, MA, other staff) in administering, tracking, and responding to PROs. A designated staff member is needed in order to ensure the steps of PRO administration are followed through for each patient. While we recommend that this responsibility falls upon a single individual that champions the collection of this data overall, other individuals, such as MAs, may be designated to act as ‘point people’ for PRO collection on a day-to-day basis.

You will want to designate in advance:

• Who schedules patients for PROs, if these are to occur on-site prior to appointment
• Who maintains list of patients that should not be offered PROs, in order to avoid scheduling them
• Who introduces PROs to the patient and sets them up with the tablet
• Who decides at point of care whether it is appropriate to administer PROs to a particular patient based on timing (e.g., late arrival, provider running behind, etc.)
• Who monitors level of completion of PROs, offers help if needed, and collects tablet
• Who coordinates clinic response if suicidality or other risk behaviors are endorsed
• Who delivers results, if paper-based delivery
• Who sanitizes tablet in between patients
• Who discusses results with patients

5.3. Create protocol: when NOT to administer PROs

We recommend setting clear protocols regarding whether to ask patients to complete PROs, taking patient-based and flow-related factors into account.

Patient-based factors

We do not recommend administering PROs during acute or urgent care visits, as in those cases the priority is to address their acute issue; these patients are more likely to be too ill or in distress to tolerate an assessment. Similarly, PLHIV with known low literacy levels, limited command of available languages, impaired cognitive functioning, impaired physical functioning that impedes answering questions, and patients historically easily agitated by questionnaires, should not be asked to self-administer PROs.

Flow-related factors

Flow-related factors should also be taken into consideration when deciding whether to administer PROs to a particular patient. Is the patient late? If so, is there enough time for the patient to complete the assessment? How busy is the clinic day? Is there sufficient private space and time for the patient to comfortably and confidentially self-administer the assessment without delaying other patients and providers? If providers are running late, is there an opportunity to administer PROs during waiting periods to help occupy time? For example, if a patient is late for their appointment and the provider is ready to see the patient and has a full schedule, it may be wiser to forego PROs for that particular patient on that day in the interest of not impeding clinic flow.

5.4. Set a standard for introducing PROs to patients

Consistent, accurate language is critical when introducing PROs to patients. We recommend that this messaging include the following attributes:

• Use a term patients will understand when referring to PROs, such as “personal health assessment” or “health questionnaire”; avoid terms like “survey”, which connotes population data
• Very brief overview of the nature of PRO content (e.g., “symptoms and health behaviors”)
• Conveyance that the information is important to care team
• Gives care team a more complete picture of overall health
• Estimated time to completion
• Provider/care team will review the results
• Answering questions is optional
• Consider providing patients a copy of results.

A sample script is below:

Hi, I’m < name >. Your provider < name > is interested in finding out a little bit more about how you’re doing before your appointment starts. This questionnaire gives us a better idea of your general health, like your health behaviors and any symptoms you might have. It takes about < x > minutes to complete. All questions are optional. Are you willing to do this today? (if yes) Great. Let us know if you get stuck and need help.

5.5. Define emergency and high-risk protocols

PRO results may identify a life-threatening health issue, such as current suicidal ideation or intimate partner violence, requiring the use of emergency protocols. PRO platforms can be configured to trigger pager alerts in real time when these or other symptoms are endorsed.

Consider which staff should receive alert for these circumstances, and how staff will respond to alerts in a timely fashion.

It is also possible to set alerts for other high-risk circumstances or behaviors, such as lack of housing or poor ART adherence which may be useful to a variety of clinic team members such as case managers, health educators, or on-site pharmacists.

Pilot

Once the workflow protocols are in place, and clinic staff trained (see Chapter 6 – Train Clinic Personnel), we strongly recommend conducting a pilot test of the PROs and their operationalization within clinic workflow prior to launching PROs with actual patients. Walk through the patient experience from start to finish. Among the areas to test in this phase:

• Wifi connectivity. Test the assessment in all room areas in which patients will be administered PROs to ensure adequate internet connection
• Responsiveness of touch-screen device
• Ease of navigating content
• Success of paging protocols when urgent issues are endorsed
• Ease of generating results and delivery to provider
• Ease of provider interpretation of results
• Anticipated patient experience. Are they being moved around during this process for other purposes, such as collection of vital signs? How easy or difficult is it to follow up with the patient? Is it clear to the PRO “point person” when the patient has finished or if they need help?
• Real vs. anticipated time burden.

5.7. Launch

We recommend a gradual approach, such as implementing PROs with a limited number of providers and their patients to start with, this allows for troubleshooting before a wider-scale launch. Check in with patients in this early phase regarding their experience of the process and solicit their insights for how to improve it. Check in with staff at all levels regularly (e.g. once after the morning shift and once at the end of the afternoon shift) for the first few days, including front desk staff, MAs, and providers to assess and troubleshoot impact on flow, and make adjustments as needed.

Case study – Graduated Approach to Implementation

… we did a graduated approach, ... we definitely had a lot of reluctant providers, … I think having the gradual approach was good because it got other people interested, … once you’re on this system you no longer have to hand out specific PRO health questionnaires, like the PHQ9 for depression to the patient, collect that and then enter that in. All of it would happen automatically through our PRO system. So, that actually increased buy-in.. and had a lot of people interested and eager to join the program.
- ePRO Manager

5.8. Resources

• ISOQOL User’s guide to implementing patient-reported outcomes assessment in clinical practice. Version 2: January 2015. Available from: https://www.isoqol.org/wp-content/uploads/2019/09/2015UsersGuide-Version2.pdf. Accessed October 2020.
• PROMIS health measures. Available from: https://www.healthmeasures.net/explore-measurement-systems/promis. Accessed October 2020.
• REDCap shared library, REDCap data collection instruments. Available from: https://redcap.vanderbilt.edu/consortium/library/search.php. Accessed October 2020.
• PCORI Patient-Reported Outcomes (PRO) Infrastructure Workshop “Integrating PROs into EHRs” Atlanta, November 19–20, 2013, Draft – October 15, 2014. Available from: https://www.pcori.org/sites/default/files/PCORI-PRO-Infrastructure-Workshop%20Report-111913.pdf. Accessed October 2020.
• The Ottawa Hospital, Patient Decision Aids. Available from: https://decisionaid.ohri.ca/azlist.html. Accessed October 2020.
• Mayo Clinic Shared Decision Making National Resource Center. Depression medication choice ‘What you should know’. Available from: https://cdn.prod-carehubs.net/n1/56fab03a15e99046/uploads/2013/11/MC5733-36rev0717_whatushouldno_WEB-1.pdf. Accessed October 2020.
• Dartmouth-Hitchcock. Decision support toolkit for primary care. Available from: https://med.dartmouth-hitchcock.org/csdm_toolkits/primary_care_toolkit.html. Accessed October 2020.
• National Learning Consortium. Shared decision making fact sheet. Available from: https://www.healthit.gov/sites/default/files/nlc_shared_decision_making_fact_sheet.pdf. Accessed October 2020.
• Agency for Healthcare Research and Quality (AHRQ). The SHARE Approach: a model for shared decision making. Available from: https://www.ahrq.gov/sites/default/files/publications/files/share-approach_factsheet.pdf. Accessed October 2020.

6. TRAIN CLINIC PERSONNEL

6.1. Initial training

A general overview of the PROs and their value will be relevant to all involved, with specific training elements relevant to the execution of individual roles.

Figure 7 shows a sample training agenda covering the topics necessary to equip staff and providers to begin administering PROs. This may be broken up into more than one training session, such as one for orienting to PROs in general, and another detailing or developing protocols; or, for training different levels of staff. In our experience, initial trainings that include all staff types work well, particularly for smaller clinics. The training becomes a dynamic and iterative process in which staff members in different roles collaboratively think through how patients administered PROs will move through their visit, refining protocols in the process.

PRO output given to providers should be easy enough to interpret that no specialized training is needed.

6.2. Ongoing Training

Consider providing refresher training or process checks as time goes on to ensure that the process is operating optimally. Options include observation by another staff member or a formal refresher session.

PRACTICAL TIP
If the clinic has an orientation manual for new hires, include a short paragraph about the PRO process

PRACTICAL TIP
Having staff complete a demonstration PRO assessment on the tablet can quickly demonstrate how easy it is, and greatly decrease hurdles.

PRACTICAL TIP
While formal refresher sessions are one option to provide ongoing training and maintain enthusiasm, we have found that annual brief presentations of key findings (e.g. % of PLHIV reporting depression, methamphetamine use, etc.) to the entire staff (as part of quarterly staff and provider joint meetings) has resulted in ownership and ongoing support not just from the providers but also from other staff members.

6.3. Resources

• eyeforpharma. Using electronic patient-reported outcomes (ePRO) for the first time: lessons learned. Available from:https://www.reutersevents.com/pharma/commercial/using-electronic-patient-reported-outcomes-epro-first-time-lessons-learned. Accessed October 2020.
• Schick-Makaroff K, Molzahn A. Strategies to use tablet computers for collection of electronic patient-reported outcomes. Health Qual Life Outcomes 2015;13:2.

7. MONITOR AND EVALUATE

Even though there is no correlation between patient satisfaction and quality of care, we all have to keep our patients satisfied. So, if patients feel like they are being heard and if their visits go better and if they get to voice their issues more completely by using a PRO, then chances are they are going to feel better about the visit. Then patient satisfaction scores will likely rise and those are also important to administrators and people who are vying for healthcare contracts. Patient satisfaction gets posted on the web and is tied to reimbursement.
- Physician/Medical Director, speaking about implementation experience

The implementation of any new process should be accompanied by a monitoring and evaluation (M&E) process to measure success and identify areas for improvement.

This section provides a simple overview of some important considerations.

7.1. Identify indicators of success

Clear and concise indicators are the basis of any effective M&E system. Two types of indicators are important to consider when assessing a PRO program (Table 8):

1. Process indicators, which provide information about the scope and execution of the process.
2. Performance indicators, which include program outcome information such as the effectiveness of service delivery.

One simple method of reviewing any indicators to use in the M&E process is to use the SMART criteria (see below). Consider each of these points when developing new indicators or revising old ones.

• Specific: The indicator should accurately describe what is intended to be measured and should not include multiple measurements in one indicator.
• Measurable: The indicator produces consistent results if obtained and tracked under the same conditions.
• Attainable: Collecting data for the indicator should be simple, straightforward, and cost-effective.
• Relevant: The indicator should be closely connected with each respective PRO outcome.
• Time-bound: The indicator should include a specific timeframe.

Examples of indicators developed using the SMART indicators are shown below in Table 9.

7.2. Determine process and timing

An M&E plan should be designed to measure progress over a program’s life span. However, year-to-year M&E strategies are common as priorities change.

Within annual cycles, data collection should occur at least once between the start and end of the program year. Frequency of collecting data is mostly dependent on the program’s cost and length i.e. longer programs, or those with more funding, can typically collect comprehensive data more frequently than shorter programs or those with less funding.

Data collection methods should be carefully considered to minimize the risk of bias arising from the method chosen. Ideally, an individual with research experience should have oversight of the proposed approach.

7.3. Implement a process of continuous quality improvement

It helped in our setting to assign one domain to an HIV doctor and an HIV nurse. Once per month during an hour, a certain outcome indicator and related process indicators are being discussed during the weekly clinical HIV meeting. Not just one champion but making everybody partly responsible.
- HIV care physician steering committee member

Continuous quality improvement (CQI) is the process of improving the program on a continuing basis. It can be described as an ongoing cycle of collecting data and using it to make decisions to gradually improve program processes.59 It therefore becomes key that a well-developed M&E plan precedes CQI action and decision-making. Several suggestions on proactively collecting feedback are below as are two examples of CQI.

• Keep lines of feedback in place
• Be ready to make changes in PRO implementation as clinic processes change and evolve
• Continue to have staff act as patients to ensure that flow and ease of use are in good shape

Figure 8 includes two examples of CQI specific to PRO implementation.

Other variables that could be observed over time for any changes include

• Changes in total appointment length
• Number of patients completing PROs
• Patient satisfaction with care
• Patient or provider perception of usefulness of PROs
• Time providers spend reviewing and integrating the PRO into the care process
• Changes in visit length
• Time spent by staff supporting PRO implementation – ongoing
• Costs and resource use.

CQI frameworks can help clinics implement and measure change at their facilities. One such framework is the Plan, Do, Study, Act (PDSA) cycle, which is a cyclical process for developing and implementing change:

• ‘Plan’ means to collect and analyze data and develop solutions to improve the program
• ‘Do’ means to implement one of the proposed solutions
• ‘Study’ means to measure any changes as the result of the proposed solution that was implemented
• ‘Act’ means to adopt the solution of standard practice, or start over.

CQI is designed to be executed quickly—and thereby reduce the time needed to test solutions through evaluation—so stakeholders can see results more quickly.⁶⁰

The CQI process aims to bring multiple stakeholders together. The integration of stakeholders ensures the appropriate expertise to gather and analyze program outcomes meaningfully, and to then suggest, implement, and evaluate any quality improvement efforts at the program and organizational level.

PRACTICAL TIP
Formal M&E plans are important in many settings to ensure ongoing success of programs and are often conducted annually. In our clinics, we have been particularly pleased with a more straightforward but ongoing approach that involves tracking completion rates. This measure is quick to generate, and easily trackable on a monthly basis. While it is a very simple measure as nursing/front desk/MA and other staff turnover, priorities can drift and it quickly demonstrates that there is a need for retraining and reminders regarding the importance of the PROs.

A second streamlined approach to M&E that has proven both extremely valuable and also very efficient is examining reasons for those who did not complete the PROs. Was it because they forgot their glasses (we now keep several pairs of non-prescription reading glasses available)? Was it because the demographic characteristics of the clinic have changed? We now offer the PRO assessment in Amharic at two of our clinics. Selecting several days every 6 months and evaluating the reasons why PLHIV seen in clinic do not complete the PROs on those days has proven to be an extremely efficient practical and useful approach to ongoing M&E.

7.4. Resources

• Minnesota Department of Health: SMART Goals Guidance
• CDC Tools: SWOT Analysis Tool, SMART Objectives Template
• Agency for Health Research and Quality (AHRQ): Quality Measures: PROs for Quality Improvement of Clinical Practice
• National Quality Forum (NQF): Patient-Reported Outcomes in Performance Measurement
• The Commonwealth Fund: Using PROs to Improve Health Care Quality
• W.K. Kellogg Foundation (WKKF): Step-by-Step Guide to Evaluation

APPENDIX 1. List of Abbreviations

APPENDIX 2. PRO timing tool

try it yourself here

APPENDIX 3. Sample stakeholder presentation

ACKNOWLEDGMENTS AND FUNDING

This Toolkit was made possible by funding from ViiV Healthcare Limited, a subsidiary of GlaxoSmithKline. Their support financed the PROgress Study, which evaluated the process of implementing electronic PROs in routine outpatient HIV care, as well as their utility, usability, acceptability, and impact. We thank the patients, providers, and research staff from two PROgress Study clinics, Midway Specialty Care in Ft. Pierce, FL, and St. Michael’s Hospital in Toronto, ON, for their rich insights and commitment to this work. We thank the staff of the Ontario HIV Treatment Network and affiliated clinics for their continued collaborative partnership in advancing the role of PROs in HIV care. We also thank the Centers for AIDS Research Network of Integrated Clinical Systems (CNICS), and the University of Washington Clinical Informatics Research Group (CIRG), for provision of resources and expertise.

Finally, we thank the PROgress Study Steering Committee, an expert panel of individuals listed below which includes patients living with HIV, HIV care providers, clinic directors, and healthcare researchers, for their oversight and guidance.

In alphabetical order
Jean Bacon, Executive Director, Ontario HIV Treatment Network, Toronto, Canada
Jeff Berry, patient advocate, USA
Heidi Crane, MD, PROgress co-Principal Investigator, HIV care provider, University of Washington, Seattle, WA, USA
Emma Fitzsimmons, PROgress Study Coordinator, University of Washington, Seattle, WA, USA
Rob Fredericksen, PhD, PROgress co-investigator, University of Washington, Seattle, WA, USA
Kevin Gough, MD, Medical Director, HIV Service Head, St. Michael’s Hospital, Toronto, Canada
Sean Hosein, patient advocate, Canada
Abigail Kroch, PhD, Director, Data and Applied Science Impact, Ontario HIV Treatment Network, Toronto, Canada
William Lober, MD, PROgress co-Principal Investigator, University of Washington, Seattle, WA, USA
Justin McReynolds, MS, Technical Program Manager, University of Washington, Seattle, WA, USA
Alex Musten, Senior Lead, Testing and Clinical Care Initiatives, Ontario HIV Treatment Network, Toronto, Canada
Edgar Overton, MD, HIV care provider, University of Alabama Hospital, Birmingham, AL, USA
Moti Ramgopal, MD, Director, Midway Specialty Care Center, Ft. Pierce, FL, USA
Duncan Short, PhD, Director, Global Implementation Science, ViiV Healthcare, Brentford, UK
Guido van den Berk, MD, HIV care provider, OLVG Hospital, Amsterdam, The Netherlands

References

1. Snyder CF, Aaronson NK, Choucair AK, et al. Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations. Qual Life Res 2012;21:1305-14.
2. Jensen RE, Rothrock NE, DeWitt EM, et al. The role of technical advances in the adoption and integration of patient-reported outcomes in clinical care. Med Care 2015;53:153-9.
3. International Society for Quality of Life Research (prepared by Chan E ET, Haywood K, Mikles S, Newton L),. Companion guide to implementing patient reported outcomes assessment in clinical practice. Available at: https://www.isoqol.org/wp-content/uploads/2019/09/ISOQOL-Companion-Guide-FINAL.pdf. Accessed October 2020.
4. Murphy EL, Collier AC, Kalish LA, et al. Highly active antiretroviral therapy decreases mortality and morbidity in patients with advanced HIV disease. Ann Intern Med 2001;135:17-26.
5. Crane HM, Crane PK, Tufano JT, et al. HIV provider documentation and actions following patient reports of at-risk behaviors and conditions when identified by a web-based point-of-care assessment. AIDS Behav 2017;21:3111-21.
6. Fredericksen RJ, Crane P, Tufano J, et al. Integrating a web-based patient assessment into primary care for HIV-infected adults. J AIDS HIV Res 2012;4:47-55.
7. Williams MV, Davis T, Parker RM, et al. The role of health literacy in patient-physician communication. Fam Med 2002;34:383-9.
8. Narayan MC. Culture's effects on pain assessment and management. Am J Nurs 2010;110:38-47. Quiz 48-39.
9. Kissinger P, Rice J, Farley T, et al. Application of computer-assisted interviews to sexual behavior research. Am J Epidemiol 1999;149:950-4.
10. Fredericksen RJ, Tufano J, Ralston J, et al. Provider perceptions of the value of same-day, electronic patient-reported measures for use in clinical HIV care. AIDS Care 2016;28:1428-33.
11. Wolfe F, Pincus T, Thompson AK, et al. The assessment of rheumatoid arthritis and the acceptability of self-report questionnaires in clinical practice. Arthritis Rheum 2003;49:59-63.
12. Stover A, Irwin DE, Chen RC, et al. Integrating patient-reported outcome measures into routine cancer care: cancer patients' and clinicians' perceptions of acceptability and value. EGEMS (Wash DC) 2015;3:1169.
13. Detmar SB, Muller MJ, Schornagel JH, et al. Health-related quality-of-life assessments and patient-physician communication: a randomized controlled trial. JAMA 2002;288:3027-34.
14. Wagner AK, Ehrenberg BL, Tran TA, et al. Patient-based health status measurement in clinical practice: a study of its impact on epilepsy patients' care. Qual Life Res 1997;6:329-41.
15. Velikova G, Booth L, Smith AB, et al. Measuring quality of life in routine oncology practice improves communication and patient well-being: a randomized controlled trial. J Clin Oncol 2004;22:714-24.
16. Taenzer P, Bultz BD, Carlson LE, et al. Impact of computerized quality of life screening on physician behaviour and patient satisfaction in lung cancer outpatients. Psychooncology 2000;9:203-13.
17. Brown RF, Butow PN, Dunn SM, et al. Promoting patient participation and shortening cancer consultations: a randomised trial. Br J Cancer 2001;85:1273-9.
18. Wasson JH, Stukel TA, Weiss JE, et al. A randomized trial of the use of patient self-assessment data to improve community practices. Eff Clin Pract 1999;2:1-10.
19. Nelson EC, Eftimovska E, Lind C, et al. Patient reported outcome measures in practice. BMJ 2015;350:g7818.
20. Chen J, Ou L & Hollis SJ. A systematic review of the impact of routine collection of patient reported outcome measures on patients, providers and health organisations in an oncologic setting. BMC Health Serv Res 2013;13:211.
21. Fanning J & McAuley E. A comparison of tablet computer and paper-based questionnaires in healthy aging research. JMIR Res Protoc 2014;3:e38.
22. Abernethy AP, Herndon JE, 2nd, Wheeler JL, et al. Improving health care efficiency and quality using tablet personal computers to collect research-quality, patient-reported data. Health Serv Res 2008;43:1975-91.
23. Fredericksen RJ, Short D, Fitzsimmons E, et al. Patient perceptions of the utility and impact of a same-day self-administered routine electronic patient-reported outcomes (PRO) assessment in HIV care in two North American clinics. 27th Annual Conference of International Society for Quality of Life Research (ISOQOL); October 19–23, 2020; Virtual conference.
24. Fredericksen RJ, Harding B, Nance RM, et al. Acceptability of electronic patient-reported outcomes (PRO) assessments among patients in HIV care. Poster 2039 presented at 26th Annual Conference of International Society for Quality of Life Research (ISOQOL); October 20–23, 2019; San Diego, CA, USA.
25. Lawrence ST, Willig JH, Crane HM, et al. Routine, self-administered, touch-screen, computer-based suicidal ideation assessment linked to automated response team notification in an HIV primary care setting. Clin Infect Dis 2010;50:1165-73.
26. Bengtson AM, Pence BW, Moore R, et al. Relationship between ever reporting depressive symptoms and all-cause mortality in a cohort of HIV-infected adults in routine care. AIDS 2017;31:1009-16.
27. Cholera R, Pence BW, Bengtson AM, et al. Mind the gap: gaps in antidepressant treatment, treatment adjustments, and outcomes among patients in routine HIV care in a multisite U.S. clinical cohort. PLoS One 2017;12:e0166435.
28. Centers for Disease Control and Prevention. Intersection of intimate partner violence and HIV in women. Available at: https://www.cdc.gov/violenceprevention/pdf/ipv/13_243567_green_aag-a.pdf?response_type=embed. Accessed October 2020.
29. Fredericksen RJ, Short D, Fitzsimmons E, et al. Provider perceptions of utility and impact of a same-day routine electronic patient reported outcomes (PRO) assessment in clinical HIV care. 27th Annual Conference of International Society for Quality of Life Research (ISOQOL); October 19–23, 2020; Virtual conference.
30. Beauclair R, Meng F, Deprez N, et al. Evaluating audio computer assisted self-interviews in urban South African communities: evidence for good suitability and reduced social desirability bias of a cross-sectional survey on sexual behaviour. BMC Med Res Methodol 2013;13:11.
31. Adebajo S, Obianwu O, Eluwa G, et al. Comparison of audio computer assisted self-interview and face-to-face interview methods in eliciting HIV-related risks among men who have sex with men and men who inject drugs in Nigeria. PLoS One 2014;9:e81981.
32. Ghanem KG, Hutton HE, Zenilman JM, et al. Audio computer assisted self interview and face to face interview modes in assessing response bias among STD clinic patients. Sex Transm Infect 2005;81:421-5.
33. Fredericksen R, Fitzsimmons E, McReynolds J, et al. Patients’ perceptions of value of a self-administered electronic patient-reported outcomes assessment on quality of their HIV care. Poster 1022 presented at 26th Annual Conference of International Society for Quality of Life Research (ISOQOL); October 20–23, 2019; San Diego, CA, USA.
34. Fenton S, Giannangelo K, Kallem C, et al. Data standards, data quality, and interoperability (updated). J AHIMA 2013;84:64-9.
35. DAMA UK Working Group on “Data Quality Dimensions". The six primary dimensions for data quality assessment: defining data quality dimensions. Available at: https://www.whitepapers.em360tech.com/wp-content/files_mf/1407250286DAMAUKDQDimensionsWhitePaperR37.pdf. Accessed October 2020.
36. Campbell N, Ali F, Finlay AY, et al. Equivalence of electronic and paper-based patient-reported outcome measures. Qual Life Res 2015;24:1949-61.
37. Meirte J, Hellemans N, Anthonissen M, et al. Benefits and disadvantages of electronic patient-reported outcome measures: systematic review. JMIR Perioper Med 2020;3:e15588.
38. Spitzer RL, Kroenke K & Williams JB. Validation and utility of a self-report version of PRIME-MD: the PHQ primary care study. Primary Care Evaluation of Mental Disorders. Patient Health Questionnaire. JAMA 1999;282:1737-44.
39. Kroenke K, Spitzer RL & Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med 2001;16:606-13.
40. Spitzer RL, Kroenke K, Williams JB, et al. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med 2006;166:1092-7.
41. Ainsworth BE, Jacobs DR, Jr. & Leon AS. Validity and reliability of self-reported physical activity status: the Lipid Research Clinics questionnaire. Med Sci Sports Exerc 1993;25:92-8.
42. Simoni JM, Kurth AE, Pearson CR, et al. Self-report measures of antiretroviral therapy adherence: a review with recommendations for HIV research and clinical management. AIDS Behav 2006;10:227-45.
43. Chesney MA, Ickovics JR, Chambers DB, et al. Self-reported adherence to antiretroviral medications among participants in HIV clinical trials: the AACTG adherence instruments. Patient Care Committee & Adherence Working Group of the Outcomes Committee of the Adult AIDS Clinical Trials Group (AACTG). AIDS Care 2000;12:255-66.
44. Lu M, Safren SA, Skolnik PR, et al. Optimal recall period and response task for self-reported HIV medication adherence. AIDS Behav 2008;12:86-94.
45. Kiechl S, Werner P, Egger G, et al. Active and passive smoking, chronic infections, and the risk of carotid atherosclerosis: prospective results from the Bruneck Study. Stroke 2002;33:2170-6.
46. Newcombe DA, Humeniuk RE & Ali R. Validation of the World Health Organization Alcohol, Smoking and Substance Involvement Screening Test (ASSIST): report of results from the Australian site. Drug Alcohol Rev 2005;24:217-26.
47. WHO ASSIST Working Group. The Alcohol, Smoking and Substance Involvement Screening Test (ASSIST): development, reliability and feasibility. Addiction 2002;97:1183-94.
48. Bradley KA, Bush KR, Epler AJ, et al. Two brief alcohol-screening tests from the Alcohol Use Disorders Identification Test (AUDIT): validation in a female Veterans Affairs patient population. Arch Intern Med 2003;163:821-9.
49. Bush K, Kivlahan DR, McDonell MB, et al. The AUDIT alcohol consumption questions (AUDIT-C): an effective brief screening test for problem drinking. Ambulatory Care Quality Improvement Project (ACQUIP). Alcohol Use Disorders Identification Test. Arch Intern Med 1998;158:1789-95.
50. Fredericksen RJ, Mayer KH, Gibbons LE, et al. Development and content validation of a patient-reported sexual risk measure for use in primary care. J Gen Intern Med 2018;33:1661-8.
51. McLellan AT, Alterman AI, Cacciola J, et al. A new measure of substance abuse treatment. Initial studies of the treatment services review. J Nerv Ment Dis 1992;180:101-10.
52. Whitney BM, Fitzsimmons EF, Jackson MK, et al. Patient perceptions and understanding of a housing status measure for use in HIV care. 27th Annual Conference of International Society for Quality of Life Research (ISOQOL); October 19–23, 2020; Virtual conference.
53. Fitzsimmons E, Loo S, Dougherty S, et al. Development and content validation of the IPV-4, a brief patient-reported measure of intimate partner violence for use in HIV care. Poster 1106 presented at 26th Annual Conference of International Society for Quality of Life Research (ISOQOL); October 20–23, 2019; San Diego, CA, USA.
54. World Health Organization. Adverse Childhood Experiences International Questionnaire (ACE-IQ). Available at: http://www.who.int/violence_injury_prevention/violence/activities/adverse_childhood_experiences/en/. Accessed October 2020.
55. Fredericksen RJ, Fitzsimmons E, Gibbons LE, et al. Development and content validation of the Multifactoral assessment of perceived social support (MAPSS), a brief, patient-reported measure of social support for use in HIV care. AIDS Care 2019;31(Suppl 1):1-9.
56. Earnshaw VA, Smith LR, Chaudoir SR, et al. HIV stigma mechanisms and well-being among PLWH: a test of the HIV stigma framework. AIDS Behav 2013;17:1785-95.
57. EuroQol Group. EuroQol: a new facility for the measurement of health-related quality of life. Health Policy 1990;16:199-208.
58. Cameron IM, Crawford JR, Lawton K, et al. Psychometric comparison of PHQ-9 and HADS for measuring depression severity in primary care. Br J Gen Pract 2008;58:32-6.
59. Minnesota Department of Health. Continuous Quality Improvement (CQI). Available at: https://www.health.state.mn.us/communities/fhv/cqi.html. Accessed October 2020.
60. Hunter S, Ebener P, Chinman M, et al. Promoting success: a getting to outcomes® guide to implementing continuous quality improvement for community service organizations. Santa Monica, CA: RAND Corporation, 2015. Available at: https://www.rand.org/pubs/tools/TL179.html. Accessed October 2020.