Marilyn Standifer Shreve is a clinical pharmacist who led California’s efforts to become the first state to implement policies and regulations that authorize electronic transmission of prescriptions (e-prescribing).

Over the last ten years, there has been much discussion within the healthcare industry about patient-centered care or improving the experience of patients. However, in a 21^st Century healthcare system, there is a need to move from patient experience and satisfaction to patient engagement and health outcomes.   How do you engage and motivate patients to take an active role in managing their treatment regimens; improving their medication-taking behavior (medication adherence); and, adopting healthy lifestyles?  Today’s Call to Action for clinicians is to employ technology to expand their voice, reach and patient engagement care beyond the walls of traditional practice settings.  The following four goals can be achieved through technology-assisted patient engagement strategies that:Improve treatment and medication adherence (patient health outcomes);

• Improve treatment and medication adherence (patient health outcomes)
• Reach large diverse patient populations and achieve better health outcomes;
• Improve patient engagement through tailored communications; and,
• Lower health care costs.

The following Patient Engagement Best Practices Resource Document was compiled by the Patient-centered Comparative Effectiveness Research Center in collaboration with Brigham Research Institute as a guide to the patient engagement process within patient-centered outcomes research (PCOR). Let us know if this resource guide is of value to you as begin to create measurable patient engagement programs.

PATIENT-CENTERED COMPARATIVE EFFECTIVENESS RESEARCH CENTER     Patient Engagement Best Practices Resource Document

 Overview

This document was compiled by the Patient-centered Comparative Effectiveness Research Center (PCERC) as a guide to the patient engagement process within patient-centered outcomes research (PCOR). For more information on engaging patients in the research process, please contact PCERC.

 

Who are the stakeholders in healthcare?

Stakeholders are individuals that are involved in or affected by something. In healthcare, the stakeholders include patients, patient family members /caregivers, clinicians, healthcare providers, policy makers, etc.

… While  this document will focus primarily on engaging patients and their caregivers in the research process, many of the concepts described are applicable for engaging other healthcare stakeholders in the research process. If you have questions about the inclusion of other stakeholders in your project, please contact PCERC for more information.

What is patient-centered outcomes research (PCOR)?

PCOR is research that evaluates the outcomes of healthcare practices and seeks to help patients and their caregivers to be more involved in their healthcare and to make more informed decisions about their health outcomes. The goal of PCOR is to provide evidence on the decisions made by all healthcare stakeholders.

PCOR also takes into account the preferences, values, and experiences of individual patients or groups of patients and looks to address questions such as: “Will this treatment improve the quality of my life? Given my lifestyle and preferences, which options are best for me? How will the implications of a particular therapy affect my family?”

What is patient engagement in research?

In the past, medical research has focused on health questions that were important to research and physicians while soliciting little input from patients and their caregivers (family members, friends that take care of them, etc.). However, more recently it has become clear that the health questions and outcomes that are most important to the academic community are not always those that are most important to patients and their caregivers. For this reason, meaningful patient engagement in the research process has been embraced and even required (depending on the project) for funding through institutions such as the Patient-Centered Outcomes Research Institute (PCORI) and the Agency for Healthcare Research and Quality (AHRQ).

Patient-engagement in research, in this context, is the “interactive, bi-directional exchange between stakeholders and researchers” in the research process in order to focus on the health questions and outcomes that are most important to patients and to improve the quality and relevancy of research. (Curtis, et al. 2012).

Why is Patient Engagement important?

Within traditional biomedical research, patients are typically research subjects. Within PCOR, it is important that researchers view patients as active and respected partners in the research process.

“Compelling reasons, both practical and ethical, support engaging patients in healthcare research. Patients are the ultimate user of healthcare research findings and the most important stakeholder” (Garces, et al. 2012). Involving patients and other stakeholders in the research process has many benefits. Patients and stakeholders bring an important perspective to the research team and can benefit research projects by:

• Articulating the outcomes that are most important to them
• Providing insight on patient decision making
• Bringing valuable “expertise” (i.e. personal experience with a condition) that researchers and physicians do not have
• Providing helpful input on language and cultural issues to make dissemination campaigns more effective

Additionally, collaborating with patients can lead to:

• Improved patient outcomes (by focusing on outcomes that are more meaningful to patients)
• Improved mutual trust
• Increased transparency and credibility of research
• Increased accountability of research organizations
• Increased patient satisfaction
• Reciprocal understanding (between researchers and patients/stakeholders)
• Improved subject recruitment and retention
• Improved study design (designs that are more consistent with meaningful patient outcomes, more culturally sensitive, )
• More relevant and applicable research findings
• More effective dissemination of research findings
• Greater and more rapid uptake and implementation of findings
• Patients having the opportunity to be involved in decisions that may impact their health/well being or that of future patients

“Pre-engagement” and the importance of starting early

One of the most important (but often overlooked) principles of the patient engagement is that it is essential to start early. Researchers cannot achieve truly meaningful and productive engagement if they wait until the last minute. Engaging patients requires getting to know the patient partners in the project, building trust, and allowing the patients to help shape the project from the very beginning. Waiting too long will negatively impact the project and it will be apparent to grant reviewers that the engagement is rushed and forced. Further, research indicates that “pre-engagement” is especially important when working with underrepresented populations.

If possible, pursue “pre-engagement” in earliest planning stages of your project. This entails building relationships with patients through focus groups or community groups related to population of interest for your study. “‘Pre-engagement’ of stakeholders builds trust and motivates meaningful engagement. Prior to implementing a PCOR study, investigators should utilize a period of ‘pre-engagement’ when recruiting research participants and partners. This allows time to assure that both partnering institutions and PCOR participants comprehend the study and have sufficient time to have their questions answered and concerns addressed. In addition, research participants often wish to discuss participation with family and friends; pre-engagement allows the potential PCOR participant to make an informed decision and feel comfortable participating so that those who choose to participate are better equipped and motivated to make meaningful contributions to PCOR. This “pre-engagement” may also apply to other phases of PCOR research involving patient engagement, such as partnering to identify and frame the research question or to disseminate findings” (University of Marylnd, 2012).

Continuous engagement

Both PCORI and the AHRQ stress that the importance of continuous patient and stakeholder engagement. This means that patient engagement at nearly step of the research process will enhance PCOR and lead to more patient-centered research. It is very clear in the literature on this subject, that engagement within one or two parts of the research process is not sufficient. Rather, researchers need to include patient and stakeholder input at every relevant stage of their research project.

“PCOR starts from the vantage point of individuals facing health decisions. Every step of the design, conduct, analysis, and dissemination of PCOR should be directed towards informing health decisions that affect outcomes that are meaningful to a specific group of patients. From the earliest phases of defining a research topic and formulating a study question; then identifying a study population and choosing interventions, comparators, and outcomes to measure; through the conduct of a study and analysis of results; and ultimately to the dissemination of research findings into clinical practice, researchers should ensure PCOR results accurately and effectively inform health decisions important to patients. This requires patient engagement throughout the research process” (Berg, et al. 2012).

How can patients be involved in my research project?

Patients and stakeholders should be involved in many different stages of PCOR. Below are some examples of how patients can help shape research projects. It is important to determine the points of engagement early on in the planning process of your projects, so that you can recruit patient partners that have the appropriate skills and expertise.

• Topic selection
• Formulating a study question
• Recruitment
• Determining consent procedures
• Identifying study population
• Identifying and choosing interventions/outcomes that are important to the population of interest (e.g. survival, function, symptoms, health-related quality of life, )
• Defining characteristics of study participants, interventions, outcomes
• Study design
• Agenda setting
• Monitoring study conduct and progress
• Data collection (through patient administered interviews)
• Data analysis
• Interpretation of findings
• Designing/suggesting plans for dissemination and implementation activities
• Translation
• Actual dissemination of research results (e.g. presentations, social media, )
• Implementing results in practice (e.g. helping to develop decision aid tools, clinical practice guidelines, )
• Evaluation

Methods for fostering engagement

Listed below are a few of the many methods of engaging patients. However, feel free to be creative and tailor the engagement to your particular project.

 

• In-person meetings
• Focus groups
• Patient and family advisory councils
• Individual patient interviews
• Phone calls/mobile messaging
• Skype conversations
• Email communication

Identifying outcomes that people in the population of interest care about and notice

In order for PCOR to be beneficial and meaningful to the people making specific health decisions, research proposals need to identify the outcomes that are important to individuals in the population of interest (e.g., survival, symptoms, health-related quality of life). Input from patients is necessary for accurately identifying these outcomes of interest.

In research proposals, these outcomes need to be defined clearly and evidence that supports the selection of these outcomes needs to be provided. This evidence should support “the selection of outcomes as meeting the criteria of ‘clinically meaningful,’ ‘patient-centered,’ and ‘relevant to decision-makers,’ such as patient and decision-maker input from meetings or surveys or published literature relevant to the question of interest. Select outcomes based on input directly elicited from patient informants, persons representative of the population of interest, either in previous studies or in the proposed research” (Berg, et al. 2012).

Things to think about when identifying patients

• Selecting the “right” patients very much depends on the goals of the project.
• Researchers should think about where the patients are with their condition/health issue. A patient that has  recently developed a condition will likely have a very different perspective than a patient that has had it for years.
• Patients that were recently diagnosed, dealing with a very difficult condition, or at a particularly intense stage of treatment may not be able to collaborate on a project.
• Research proposals need to describe a clear plan to ensure the representativeness of participants. Patient partners/participants should be representative of the patient population facing the health decision of interest.

Qualities to look for in patient partners

• They should have personal experience with the condition/health issue in question and the range of treatment options
• If they do not have personal experience with the condition/health issue in question, they should be knowledgeable about it and the range of treatment options
• They should be similar to the population/community that the study results are intended to affect
• They need to be objective and not let their personal experiences affect their representation of other patients
• They cannot have conflicts of interested that prevent them from participating in the research process—financial (i.e. they cannot receive an honorarium) or other (i.e. advocacy group responsibilities). Researchers should ask about any such conflicts of interest prior to
• It is beneficial if they are comfortable speaking and sharing their opinions/views in a group setting (if on focus groups)
• It is beneficial if they are comfortable interacting with other patients

Practical advice for recruiting patients

• Based on the requirements, expectations, and roles of the project, researchers should develop a description of the experience needed
• Outline the training requirements, expected roles, and expectations for the patients prior to patient recruitment. It is then important to clearly share this information with patients and to also highlight the roles that patients will not play in the research
• The process used to select and recruit patients should be balanced and inclusive in terms of ethnicity, age, gender, condition burden, and socioeconomic
• The purpose of the research study, the objectives for patient participation, and patient compensation should also be clearly outlined
• Provide patient representatives with a clear timeline and a breakdown of the anticipated time commitment for each stage of the project

• The timeline for patients receiving compensation is also very important. Partners Healthcare can be very slow in processing the patient honorariums and researchers have indicated in the past that some patients became angry and placed frustrated phone calls to the PIs when they had not received compensation several weeks after the project ended. Clarify this timeline with patients before they join the
• It is very important that potential patients understand what the research is about. Go over the goals of the project several times with each
• Patients should be informed that the recruitment is bi-directional and they should be encouraged to think about it carefully and ask questions so that they can make an informed
• Researchers should partner with several patients (more than one or two) on their project to ensure that there is sufficient representation of the patient
• Use a range of settings, techniques to recruit

Strategies for identifying and recruiting appropriate patient representatives

Strategies for recruiting patient partners/participants should vary depending on the project. Below are some of the avenues for identify patient partners/participants (including individuals from hard-to-reach populations)

• Hospital/university academic departments
• Disease specific social networks
• Patient groups (e.g. advocacy groups, disease societies, )
• Claims data
• Medical registries
• Local clinics and hospitals
• Community centers
• Community housing centers
• Community organizations (e.g. Salvation Army, Big Brothers Big Sisters, Veterans Affairs, )
• Community meetings
• Health Fairs
• Local spots (e.g. churches/youth ministries, schools, libraries, recreation centers, soup kitchen/food pantries, transitional housing, hair dressers, barber shops, grocery stores, )
• Community outreach through employers, community organizations,
• Email via list serves
• Emergency rooms/Urgent care
• Bulletin boards (e.g. grocery stores, Community centers, )
• Billboards (especially those near hospitals/health centers)
• Magazines
• Community papers
• Press releases to radio, newspaper, newsletters
• Social media (e.g. Facebook, Twitter)

Recruiting patients, partners/partcipants

• Have something to draw them to you (i.e. food) and open up a conversation
• Work with cultural groups similar to the patients
• Use appropriate language
• Re-assure them that their personal information will not be shared
• Be clear and straight to the point when telling them about the study (the project, the expectations, )
• Have incentives
• Ensure that the consent form is written using language that is understandable to them

• Avoid long consent forms, go over forms with them in person
• Make sure they understand the research project and expectations. Ask open ended questions or have them explain it back to you (Teach-Back method)

Additional BWH resources for identification, recruitment, and engagement of patients

In addition to the avenues for identifying and recruiting patients that are provided above, BWH has additional resources to help you identify and recruit patient partners for your PCOR research project.

The BWH Bretholtz Center for Patients and Families is a comprehensive resource center that provides support and information to BWH patients and families while they receive care at the hospital. In addition, the Center works with numerous Patient and Family Advisory Councils (PFACs) at BWH and has helped connect numerous PCOR researchers with patient partners. If you are interested in working with this Center or creating/attending a PFAC, please contact Maureen Fagan, the Executive Director for the Center.

The BWH Center for Community Health and Health Equity (CCHHE) is committed to “advancing systems of care, research, and community programs that elevate the health status of the communities [they] serve”. These neighboring communities include Mission Hill, Roxbury, Jamaica Plain, Mattapan, and Dorchester. The CCHHE has partnered with these communities to address real life issues including health disparities, access to care for vulnerable populations, and health awareness. The CCHHE has numerous relationships within the local community and BWH researchers can work with this Center to more effectively engage these communities and the hard- to-reach subpopulations within them. If you are interested in working with the CCHHE’s community partners to engage local patients, please email Wanda McClain, the CCHHE’s Executive Director.

Engaging patients from hard-to-reach communities

One of the most important, yet challenging requirements of PCOR is engaging patients from hard-to-reach communities within the research process. These patients include individuals “historically underrepresented in health care research such as those with multiple disease conditions, low literacy, low socioeconomic status, or poor health care access, as well as racial and ethnic minority groups and people living in rural areas” (Berg, et al. 2012). Due to barriers such as distance and socioeconomic status, this population does not typically participate in research. However, in order to effectively incorporate the values, beliefs, opinions, and health outcomes of these groups, it is very important to engage patients from these communities in the PCOR research process. “With greater involvement of patients from all walks of life, results would be more applicable to the entire spectrum of Americans and results would be implemented more rapidly across healthcare delivery systems” (University of Maryland, 2012).

Advice for Engaging Hard-to-Reach Patients

• Partner with individuals, associations, and groups that already have a network within these communities
• Go where they are located. Meet in places where people
• Utilize media (flyers, billboards, advertising in public transportation, social media, radio, TV, )
• Pre-engage to build relationships within these communities and then maintain these relationships after the project is over. Also, make a significant effort to give back to these
• Use plain language (the terms “research” and “outcomes” are unfamiliar to many patients in these communities)
• Think about how questions to these patient partners/participants is phrased
• Be flexible
• Multiple literature reviews indicated that patients may become frustrated if asked repeated questions
• Understand that some patients in these communities do now feel comfortable disclosing medical information openly
• Understand that trust is extremely important to these patients
• Be very clear about the expectations and goals of the project. Make sure the patients being recruited fully understand both of these.

• Clearly address how you will identify, select, recruit, and retain these patients in your proposal

patients

How to support and train patient representatives

Training

• All patients and researchers should receive adequate training for participation in a patient-centered research project
• Patient training should be tailored for the particular role that they will have and take into account their level of knowledge/experience. They should also be trained for research participation in general as this may be a foreign concept for them
• Provide both technical training as well as interpersonal, communication, and teamwork training for patients
• The term “research” often has negative connotations, especially in hard-to-reach populations. Provide education on what exactly is meant by “research”.
• Case studies can be effect if used in the training process
• Training materials should be written for a lay audience and not be too long in order to not overwhelm patients
• Researchers should be trained on the different roles of patients in the research process as well as how to effectively engage patients

Example Training Material

• Online Course: Understanding evidence-based healthcare: a foundation for action
• New Standards for Plain Language Summaries

Supporting Patients

• Provide no-strings-attached compensation (an honorarium, gift card, etc.) and/or support for travel to remove financial barriers for patients
• Think about things such as food, travel, parking, childcare,
• Include an adequate budget for necessary support of these activities in the research proposal
• Foster equality by ensuring that compensation rates for patients and researchers are comparable
• There is no recommended “rate” for compensation in patient-centered studies. This should vary based on the time commitments, amount of responsibility, length of the project,
• Past PCOR projects that we are aware of have varied compensation rates/methods from $50/$100 gifts cards to several hundred dollar honorariums to hourly rates of $25-$50.
• Provide materials for patients well in advance of any activity

Dissemination and Implementation

• If possible, patients should play an active role in the dissemination process
• For each patient partner/participant, the dissemination method should be personalized based on the patient’s characteristics and
• Avoid technical jargon when disseminating to lay audiences
• Provide clear guidance on patient roles and responsibilities within the group and provide additional dissemination training for patients
• Implement your dissemination plan in a timely manner once the research concludes
• Follow up with patient partners/participants in a research project and their communities/organizations with updates after the study is complete. This will promote meaningful participation in future
• If possible and appropriate, provide a celebration at the completion of the study
• For more information on dissemination of research results, please see the PCERC Research Dissemination Resource Document

Evaluation

Evaluating the engagement process as well as the dissemination process is extremely important within PCOR projects. The evaluation process should capture both the researchers and patient partners/participants perspectives and should be made available in order to benefit future PCOR projects.

Principles for effective patient engagement

Building trust

“The existence of trust is key to motivating partners and communities to participate and to enrolling patients in PCOR. Building trust is not a step in the patient engagement process, but rather an overarching paradigm that needs to be ubiquitous throughout PCOR efforts” (University of Maryland, 2012). In order to build trust, researchers must:

• Speak the same language (i.e. use “lay” language)
• Be flexible
• Be genuine and consistent
• Remember that “research” is an emotionally charged word
• Be honest. You may have to overcome patients’ mistrust of researchers. If you say you are going to do something, follow
• Start early with “pre-engagement”. Trust cannot be built overnight. Integrate yourself into existing community networks. Trust is one of the key factors in the decision of whether or not to participate in the research process for individuals from hard-to-reach groups
• Be culturally sensitive
• Be non-judgmental and treat everyone the same
• Occasionally call and check on participates
• Listen to people, understand that sometimes participants are just looking for someone to talk to

Communication

Effective communication is necessary for creating a constructive environment and eliciting patient perspectives. Strategies for productive communication with patients include:

• Communicate early and often
• Communicate in a culturally and linguistically appropriate manner
• Keep questions simple and straightforward
• Have a single contact person on the research team
• Ask questions in a non-judgmental manner
• Actively listen to patient responses. Read body language and think about non-verbal ways to build trust
• View patient partners/participants a people not patients in order to remove the stigma that having a condition/illness may bring
• Show the utmost respect for every member of the research team at all times. Never talk down to any members of the
• Provide constructive feedback to affirm that research partners/participants are contributing to the project
• Encourage patients to share their opinions and provide adequate time for them to participate
• Build relationships with patient partners/participants by providing opportunities to meet outside of schedule activities

Equality

Fostering a sense of equality is essential for effective PCOR as it helps patients feel that they are equal contributors to the project. Some strategies for promoting such equality are:

• Eliminating degrees from nametags
• Using first names for all researchers and patients rather than “Dr…”
• Offering equal compensation to all participants (patients and researchers)
• Creating opportunities for researchers to connect outside of the research setting
• Using plain language for descriptions of key concepts, the research project, and when recording summaries of calls or meetings involving patient representatives

Transparency

• Create transparent processes

• Provide clear rules for engagement within group settings
• Establish formal processes for voting
• Establish clear processes for resolving conflict

Important considerations when engaging patients in PCOR

• To effectively engage patients, you must start early and engage them frequently
• Select patients that are representative of the patient population that you are studying
• It is important to ensure that the patients truly understand what the research project is about
• Clarify the roles, relationships, workload, and incentives at the beginning of the
• Trust is crucial when engaging patients. Work hard to build trust between the researchers and patient
• It is important to give back to the participants in your research project (e.g. follow up and provide the results of the study and, if possible, go into the community that you worked with to disseminate this information)
• It is important to respect patient privacy
• It is important to be culturally sensitive
• It is important to ensure that you budget adequate resources for the training (for both patients and researches), facilitation, and support of the engagement. Include these efforts in the research
• There is no “one size fits all” method when you are engaging patients
• Be conscious of the burden (time and effort) that the engagement puts on patients, their caregivers, and the research team. Ensure that what you ask of patients is
• Always think about both the patients and their caregivers. What is important to the patients may be different that what is important to their
• Researchers must respect the patient partners, treat them as equal members of the team, and value their opinions
• Do your research before engaging patients and learn from the experiences of past researchers. There are many articles and case studies on meaningful patient engagement

Barriers to effective patient engagement

• Time constraints for researchers due to starting too late
• Time constraints for patients due to projects being too burdensome on patients (e.g. too many meetings, other time commitments)
• It is recommended that activities be scheduled outside of normal of normal work hours to reduce the effect that the research project has on patients other responsibilities
• Researcher bias in indentifying and selecting patient partners
• Not having sufficient expertise
• Tokenistic engagement focused on “checking off a box” rather than meaningfully engaging the patients

• Lack of funding for effective engagement
• Insufficient incentive for patient partners/participants (i.e. no compensation/too little compensation)
• Unclear expectations for patients
• Perceived inequality between patient and researcher participants
• Poor communication
• Patient lack of knowledge/experience in subject area

Challenges you may encounter within patient engagement

Despite the advantages of patient engagement, there are challenges to engaging patients. It is important to think about the challenges that you may encounter in your particular project, so that you can overcome these obstacles.

• Patient engagement is likely to increase the cost and time of your
• As engaging patients in the research process is fairly new, the best practices are likely to continue to evolve as more research on this subject is
• Some patients do not want to be engaged despite the approach that you
• Some patients may be angry toward and distrustful of healthcare providers

Patient engagement checklist

• Did I start the patient-engagement process very early on?
• Have I effectively engaged individuals that are representative of the population of interest, including those from hard-to-reach communities?
• Have I engaged multiple patient partners at various points of the research process?
• Am I requesting adequate resources to support the engagement process (i.e. recruiting, training, facilitating communication, )?
• Am I providing support and compensation to the patient engaged in my project to enable them to be active partners in research?
• Am I utilizing the skills/networks of my patients for the dissemination of my research results?
• Did I seek out patient engagement expertise to help me with the process?
• Did I conduct exit interviews with my patient partners to solicit what they found to be helpful/unhelpful, ?
• Is my project transparent?
• Did I evaluate my engagement?

Patient engagement expertise

For those that are new to the patient-engagement process, PCERC can provide additional advice on patient-engagement before you start your project, help facilitate the identification of patient partners, and provide guidance throughout your project. Further, we can connect your research team to patient-engagement experts at BWH who may be interested in providing either one-time consulting or joining your project as a patient-engagement advisor. Please email PCERC for more information.

Additional advice for writing your research proposal

• Be sure that your proposal identifies all relevant populations whose health decisions will be potentially affected by your
• Recruitment should involve individuals’ representative of the full spectrum of the populations of
• Ensure that the description of your approach and plan (process, method, and outcomes) for patient engagement is clear.
• Clearly explain the selection process for the engaged patients, what the engagement process looked like, how feedback was obtained, and what impact this had on the patients you worked
• Fully describe how participants are identified, selected, recruited, enrolled, and retained in the study to address/reduce potential impact of selection bias (Berg, et al. 2012)
• If applicable, describe how participants from hard-to-reach or unique populations identified, selected, recruited, enrolled, and retained
• Fully describe what you have done to maximize adherence to agreed-on enrollment practices (Berg, et al. 2012)
• Clearly describe your methods used to ensure unbiased data collection from patient partners/participants
• Describe how you will adhere to agreed upon enrollment practices
• If the patients with the condition of interest are the best source of information, use patient-reported outcomes
• Authorship credit needs to be given to all individuals closely involved in the process (despite their background)

 

Appendix A: References

1. Curtis, Pam, MS, Samantha Slaughter-Mason, MPH, Aasta Thielke, MPH, Cathy Gordon, MPH, and Catherine Pettinari, PhD. PCORI Expert Interviews Projects. Rep. Patient-Centered Outcomes Research Institute, n.d. 30 Aug. 2013. http://pcori.org/assets/pdfs/Expert%20Interviews%20Part%201.pdf, http://pcori.org/assets/pdfs/Expert%20Interviews%20Part%202.pdf, http://pcori.org/assets/pdfs/Expert%20Interviews%20Part%203.pdf, http://pcori.org/assets/pdfs/e- Expert%20Interviews%20Part%204.pdf
2. Garces, Juan Pablo D., Gabriela J. Prutsky Lopez, Zhen Wang, Tariq A. Elraiyah, Mahammed Nabhan, Jaun Brito Campana…M. H. Murad. 1 Eliciting Patient Perspective in Patient-Centered Outcomes Research: A Meta Narrative Systematic Review. Rep. Patient-Centered Outcomes Research Institute, 2012. Web. 30 Aug. 2013. http://pcori.org/assets/Eliciting-Patient-Perspective-in-Patient-Centered-Outcomes-Research-A-Meta-Narrative- Systematic-Review1.pdf
3. Bonoff, Amy, et al. Supporting Patient and Consumer Involvement in Patient Centered Outcomes Research. 2011. MS. Agency for Healthcare Research and Quality Community Forum, Washington C. http://effectivehealthcare.ahrq.gov/tasks/sites/ehc/assets/File/Consumer-Involvement-in-Patient-Centered- Outcomes-Research.pdf
4. Berg, Alfred, David Flum, Sherine Gabriel, and Sharon-Lise Normand. Draft Methodology Report: “Our Questions, Our Decisions: Standards for Patient-centered Outcomes Research” Ed. Mark Helfand. Patient- Centered Outcomes Research Institute, 23 July 2012. Web. 22 May 2013. http://pcori.org/assets/MethodologyReport-Comment.pdf
5. 1 Workshop on Incorporating the Patient Perspective into Patient-Centered Outcomes Research. 7 Mar. Hosted by the Patient-Centeredness Work Group of the Patient-Centered Outcomes Research Institute Methodology Committee. Baltimore, MD. http://pcori.org/assets/Workshop-on-Incorporating-the-Patient-

Perspective-into-PCOR1.pdf

6. Workshop on Incorporating the Patient Perspective into Patient-Centered Outcomes Research. 7 Mar. Hosted by the Patient-Centeredness Work Group of the Patient-Centered Outcomes Research Institute Methodology Committee. Baltimore, MD. http://pcori.org/assets/Workshop-on-Incorporating-the-Patient- Perspective-into-PCOR1.pdf
7. Integrating Patients’ Voices in Study Design Elemen Ts with a Focus on Hard-to-reach Populations. University of Maryland, n.d. Web. 28 Aug. 2013. http://pcori.org/assets/pdfs/Integrating%20Patients%20Voices.pdf
8. Deverka PA, Lavallee DM, Desai PJ, et al. Stakeholder Participation in Comparative Effectiveness Research: Defining a Framework for Effective Engagement. Journal of Comparative Effective Research.In
9. Mullins, C. Daniel, PhD, Abdulla M. Abdulhalim, BSPharm, and Danielle C. Lavallee, PhD. “Continuous Patient Engagement in Comparative Effectiveness Research.” The Journal of the American Medical Association 15 (2012): 1587-588. Web. 29 Aug. 2013. http://jama.jamanetwork.com/article.aspx?articleid=1148160