Difference between revisions of "Data Validation and Quality Assessment"

From PHUSE Wiki
Jump to: navigation, search
m (Reverted edits by DanBoisvert (talk) to last revision by Cdecker)
Line 1: Line 1:
#REDIRECT [[Working Group:Data Validation and Quality Assessment]]
+
{{RightTOC}}
 +
= Mission =
 +
This working group will focus on collaborating to develop a robust process to rapidly validate and assess data quality as data moves through the product life cycle across both industry and regulatory review. The group will discuss current pains and potential solutions around topics such as the current data validation rules as well as the CDISC Data Validation Project, development and implementation of tools, terminology, and improving the quality of the data to support analytical needs.
 +
 
 +
==Background==
 +
FDA recently has embarked on an aggressive effort to modernize the review process that will rely on high quality standardized data to streamline acquisition, analysis, storage and reporting. This new process takes a “reviewer-centered” approach where improved data quality, accessibility and predictability will allow reviewers more time to carry out complex analyses, ask in-depth questions and address late-emerging issues.
 +
To support the transition to a more advanced review process, FDA has initiated the Data Validation Project. The objective of the project is to evaluate existing validation rules, craft new validation rules, and implement tools to enable systematic, automatic and regular evaluation of data quality and standards conformance at the time of product submission. The project will inform reviewers of characteristics of submitted data, allow for cross-submission assessment of potential problem areas,  and help generate a better understanding of several key factors related to data standards adoption, including the rate of uptake of standardized data by sponsors, the degree of adherence to the specified standards, and the ability of reviewers to use standard-specific advanced review methods/tools.
 +
 
 +
== Leadership ==
 +
 
 +
{| class="wikitable"
 +
|-
 +
! Name !! Role !! Organization!! E-mail
 +
|-
 +
| Max Kanevsky || Industry Co-lead ||OpenCDISC||  mkanevsky@pinnacle21.net
 +
|-
 +
| Hany Aboutaleb|| Industry Co-lead|| Biogen Idec||hany.aboutaleb@biogenidec.com
 +
|-
 +
| Amy Malla|| FDA Co-lead ||FDA - CBER || Amy.Malla@fda.hhs.gov
 +
|-
 +
| Armando oliva || FDA Co-lead || FDA - CBER ||armando.oliva@fda.hhs.gov
 +
|-
 +
| Mitra Rocca || FDA Co-lead || FDA - CBER || Mitra.Rocca@fda.hhs.gov
 +
|-
 +
| Charles Cooper || Steering Committee Liaison || FDA - CBER ||Chuck.Cooper@fda.hhs.gov
 +
|-
 +
| Susan McCune || Steering Committee Liaison || FDA-HHS ||Susan.McCune@fda.hhs.gov
 +
 
 +
|}
 +
 
 +
== Conference Calls and Minutes ==
 +
<!-- Add links to your Minutes pages by creating the page from the box and adding the name here. -->
 +
 
 +
[[WG1 DDMONYYYY]]
 +
{{#tag:inputbox |
 +
type=create
 +
buttonlabel=Create new Minutes
 +
preload=Template:WG1Minutes
 +
}}
 +
 
 +
== Approach ==
 +
The success of FDA initiatives depends on close collaboration with industry stakeholders including standards development organizations, sponsors, and academia. The approach for this working group is to provide a platform enabling open communication between FDA and industry to share ideas, experiences and expectations, and to facilitate joint development of validation rules and processes to assess data quality and standards compliance throughout the product life cycle.
 +
 
 +
==Scope==
 +
The workgroup will initially focus on the following 3 projects:
 +
 
 +
Assess and improve existing validation rules
 +
 
 +
<font color="#0097FF">'''Assess and improve existing validation rules'''</font><br />
 +
 
 +
The existing validation rules for assessing data quality and conformance with CDISC standards have been developed through 5 years of collective efforts of FDA, CDISC, OpenCDISC, and other organizations. As FDA understanding of standards and the definition of valid, high quality data has evolved, it’s time to re-assess these rules. The goal is to evaluate the effectiveness of existing rules, provide better categorization, descriptions, and rationale to assist sponsors in preparation of data for submissions. The effort to improve validation rules has already been ongoing as part of the OpenCDISC community and most recently the CDISC Advisory Board (CAB) Validation project and ADaM Validation sub-team. The goal of this workgroup is not to replicate their effort, but instead to support and enable collaboration to meet the common goals of FDA and the industry.
 +
 
 +
<font color="#0097FF">'''Develop guidelines for validation rule developers'''</font><br />
 +
 
 +
There are currently over 700 validation rules spanning SDTM, ADaM, SEND, and define.xml standards. However, with FDA Data Validation project and CDER’s initiative to develop 50+ therapeutic area specific standards, the number of rules and teams developing them are expected to grow significantly. Therefore, one of the working group’s top priorities is to develop guidelines to assist rule and domain developers. These guidelines should provide instructions, best practices, and examples for how to define, organize, and document validation rules to ensure consistency and compatibility with FDA initiatives.
 +
 
 +
<font color="#0097FF">'''Develop and support change management process'''</font><br />
 +
 
 +
With evolving data standards and active efforts to improve and develop new validation rules for data quality and standards compliance assessment, it’s important to have a flexible change management process.  The goal of this workgroup is to develop a process that provides a forum for industry to share experiences, discuss validation issues, and propose rule refinements. This process should result in actionable advice to assist FDA change management board to quickly update validation criteria to meet the needs of the review process.
 +
 
 +
==Agenda==
 +
 
 +
'''Monday, March 19th'''
 +
 
 +
<font color="#0097FF">'''Session 1 (Day 1, 1:00 - 2:30) – FDA’s Vision for Data Validation and Quality Assessment (1.5 hr)'''</font><br />
 +
 
 +
* Discuss workgroup’s logistics   
 +
::o  Frequency of meetings
 +
::o  Type of Meetings (Teleconference, Face-to-Face)
 +
::o  Recruiting of additional volunteers             
 +
::o  Communications 
 +
 +
* FDA's statements of vision and objectives for the conference and workgroup
 +
* Introduction to data validation and quality assessment
 +
::o  Importance for modernization of review process
 +
::o  Role of data standards
 +
* Overview of FDA’s data validation and quality initiatives
 +
::o  CBER/CDER Change Coordination Group
 +
::o  CDER Data Validation Project
 +
::o  FDA collaboration with industry (OpenCDISC, CAB Validation Project, etc.)
 +
*Review of workgroup goals
 +
::o  Develop guidelines for validation rule developers
 +
::o  Assess and improve existing validation rules
 +
* Design and support change management process
 +
 
 +
 
 +
* '''Break out into groups'''
 +
 
 +
<font color="#0097FF">'''Session  2 (Day 1, 3:00 - 4:30) and Session 3 (Day 2, 8:30 - 10:30) – Break Out Groups (3.5 hr)'''</font><br />
 +
 
 +
 
 +
'''Group 1 – Develop Guidelines for Validation Rule Developers (Mitra)'''
 +
 
 +
* Discuss key validation concepts
 +
::o  Standards Compliance vs. Data Quality
 +
::o  Compliant vs. Useful
 +
* Review existing approaches and common practices
 +
::o  Summary from various development teams (OpenCDISC, HL7, CAB Validation Project, and ADaM Validation sub-group)
 +
::o  What guidelines are being followed by each team?
 +
::o  How does each team gathers metrics and evaluates rule effectiveness?
 +
* Discuss and propose guidelines for
 +
::o  Rule numbering
 +
::o  Messages and Descriptions
 +
::o  Categorization
 +
::o  Severity
 +
•        Evaluate specific examples
 +
o  Review common rules from SDTM, Define.xml, and ADaM
 +
o  How can they be improved?
 +
•        Practice defining new rules
 +
o  Try to create a few new checks for SDTM Amendment 1
 +
* Summarize groups findings
 +
 
 +
'''Group 2 – Assess and Improve Existing Validation Rules (Amy)'''
 +
 
 +
* Discuss general issues with existing validation rules
 +
::o  What are common and recurring issues?
 +
::o  What  does it mean to pass or fail (not black and white)?
 +
::o  What are the main contributing factors?
 +
::::1.      Bad or incorrectly implemented rule
 +
::::2.      Unclear or conflicting underlying standards
 +
::::3.      Lack of understanding
 +
* Review and evaluate Top 20 SDTM and Define.xml issues from CBER/CDER
 +
::o  Review metrics from CBER and CDER
 +
::o  Evaluate each rule to determine contributing factors
 +
::o  Propose changes to rule definition or underlying standards
 +
* Summarize groups findings
 +
 
 +
'''Group 3 – Design Change Management Process (Armando)'''
 +
 
 +
* Propose and discuss Change Management Process
 +
::o  What are the possible approaches to change management?
 +
::o  What is the role of workgroup members?
 +
::o  What is the role of CBER/CDER CCG?
 +
::o  What should be the makeup of the Change Management Board?
 +
::o  How do we evaluate and propose changes to existing rules or new rules?
 +
::o  What is are the details of the approval process (review periods, voting, etc.)?
 +
::o  What is a reasonable release schedule?
 +
* Summarize groups findings
 +
 
 +
 
 +
'''Tuesday, March 20th'''
 +
 
 +
 
 +
<font color="#0097FF">'''Session 4 (Day 2, 11:00 - 12:00) – Wrap-up and Next Steps (1 hr)'''</font><br />
 +
 
 +
* Review the findings of break out groups
 +
* Capture the list of items for further discussion
 +
* Prepare report-out

Revision as of 12:17, 21 March 2012


Mission

This working group will focus on collaborating to develop a robust process to rapidly validate and assess data quality as data moves through the product life cycle across both industry and regulatory review. The group will discuss current pains and potential solutions around topics such as the current data validation rules as well as the CDISC Data Validation Project, development and implementation of tools, terminology, and improving the quality of the data to support analytical needs.

Background

FDA recently has embarked on an aggressive effort to modernize the review process that will rely on high quality standardized data to streamline acquisition, analysis, storage and reporting. This new process takes a “reviewer-centered” approach where improved data quality, accessibility and predictability will allow reviewers more time to carry out complex analyses, ask in-depth questions and address late-emerging issues. To support the transition to a more advanced review process, FDA has initiated the Data Validation Project. The objective of the project is to evaluate existing validation rules, craft new validation rules, and implement tools to enable systematic, automatic and regular evaluation of data quality and standards conformance at the time of product submission. The project will inform reviewers of characteristics of submitted data, allow for cross-submission assessment of potential problem areas, and help generate a better understanding of several key factors related to data standards adoption, including the rate of uptake of standardized data by sponsors, the degree of adherence to the specified standards, and the ability of reviewers to use standard-specific advanced review methods/tools.

Leadership

Name Role Organization E-mail
Max Kanevsky Industry Co-lead OpenCDISC mkanevsky@pinnacle21.net
Hany Aboutaleb Industry Co-lead Biogen Idec hany.aboutaleb@biogenidec.com
Amy Malla FDA Co-lead FDA - CBER Amy.Malla@fda.hhs.gov
Armando oliva FDA Co-lead FDA - CBER armando.oliva@fda.hhs.gov
Mitra Rocca FDA Co-lead FDA - CBER Mitra.Rocca@fda.hhs.gov
Charles Cooper Steering Committee Liaison FDA - CBER Chuck.Cooper@fda.hhs.gov
Susan McCune Steering Committee Liaison FDA-HHS Susan.McCune@fda.hhs.gov

Conference Calls and Minutes

WG1 DDMONYYYY


Approach

The success of FDA initiatives depends on close collaboration with industry stakeholders including standards development organizations, sponsors, and academia. The approach for this working group is to provide a platform enabling open communication between FDA and industry to share ideas, experiences and expectations, and to facilitate joint development of validation rules and processes to assess data quality and standards compliance throughout the product life cycle.

Scope

The workgroup will initially focus on the following 3 projects:

Assess and improve existing validation rules

Assess and improve existing validation rules

The existing validation rules for assessing data quality and conformance with CDISC standards have been developed through 5 years of collective efforts of FDA, CDISC, OpenCDISC, and other organizations. As FDA understanding of standards and the definition of valid, high quality data has evolved, it’s time to re-assess these rules. The goal is to evaluate the effectiveness of existing rules, provide better categorization, descriptions, and rationale to assist sponsors in preparation of data for submissions. The effort to improve validation rules has already been ongoing as part of the OpenCDISC community and most recently the CDISC Advisory Board (CAB) Validation project and ADaM Validation sub-team. The goal of this workgroup is not to replicate their effort, but instead to support and enable collaboration to meet the common goals of FDA and the industry.

Develop guidelines for validation rule developers

There are currently over 700 validation rules spanning SDTM, ADaM, SEND, and define.xml standards. However, with FDA Data Validation project and CDER’s initiative to develop 50+ therapeutic area specific standards, the number of rules and teams developing them are expected to grow significantly. Therefore, one of the working group’s top priorities is to develop guidelines to assist rule and domain developers. These guidelines should provide instructions, best practices, and examples for how to define, organize, and document validation rules to ensure consistency and compatibility with FDA initiatives.

Develop and support change management process

With evolving data standards and active efforts to improve and develop new validation rules for data quality and standards compliance assessment, it’s important to have a flexible change management process. The goal of this workgroup is to develop a process that provides a forum for industry to share experiences, discuss validation issues, and propose rule refinements. This process should result in actionable advice to assist FDA change management board to quickly update validation criteria to meet the needs of the review process.

Agenda

Monday, March 19th

Session 1 (Day 1, 1:00 - 2:30) – FDA’s Vision for Data Validation and Quality Assessment (1.5 hr)

  • Discuss workgroup’s logistics
o Frequency of meetings
o Type of Meetings (Teleconference, Face-to-Face)
o Recruiting of additional volunteers
o Communications
  • FDA's statements of vision and objectives for the conference and workgroup
  • Introduction to data validation and quality assessment
o Importance for modernization of review process
o Role of data standards
  • Overview of FDA’s data validation and quality initiatives
o CBER/CDER Change Coordination Group
o CDER Data Validation Project
o FDA collaboration with industry (OpenCDISC, CAB Validation Project, etc.)
  • Review of workgroup goals
o Develop guidelines for validation rule developers
o Assess and improve existing validation rules
  • Design and support change management process


  • Break out into groups

Session 2 (Day 1, 3:00 - 4:30) and Session 3 (Day 2, 8:30 - 10:30) – Break Out Groups (3.5 hr)


Group 1 – Develop Guidelines for Validation Rule Developers (Mitra)

  • Discuss key validation concepts
o Standards Compliance vs. Data Quality
o Compliant vs. Useful
  • Review existing approaches and common practices
o Summary from various development teams (OpenCDISC, HL7, CAB Validation Project, and ADaM Validation sub-group)
o What guidelines are being followed by each team?
o How does each team gathers metrics and evaluates rule effectiveness?
  • Discuss and propose guidelines for
o Rule numbering
o Messages and Descriptions
o Categorization
o Severity

• Evaluate specific examples o Review common rules from SDTM, Define.xml, and ADaM o How can they be improved? • Practice defining new rules o Try to create a few new checks for SDTM Amendment 1

  • Summarize groups findings

Group 2 – Assess and Improve Existing Validation Rules (Amy)

  • Discuss general issues with existing validation rules
o What are common and recurring issues?
o What does it mean to pass or fail (not black and white)?
o What are the main contributing factors?
1. Bad or incorrectly implemented rule
2. Unclear or conflicting underlying standards
3. Lack of understanding
  • Review and evaluate Top 20 SDTM and Define.xml issues from CBER/CDER
o Review metrics from CBER and CDER
o Evaluate each rule to determine contributing factors
o Propose changes to rule definition or underlying standards
  • Summarize groups findings

Group 3 – Design Change Management Process (Armando)

  • Propose and discuss Change Management Process
o What are the possible approaches to change management?
o What is the role of workgroup members?
o What is the role of CBER/CDER CCG?
o What should be the makeup of the Change Management Board?
o How do we evaluate and propose changes to existing rules or new rules?
o What is are the details of the approval process (review periods, voting, etc.)?
o What is a reasonable release schedule?
  • Summarize groups findings


Tuesday, March 20th


Session 4 (Day 2, 11:00 - 12:00) – Wrap-up and Next Steps (1 hr)

  • Review the findings of break out groups
  • Capture the list of items for further discussion
  • Prepare report-out