Themes that Cut Across Disciplines: Quality Control and Research MethodsOverviewWhat do scientists have in common?History of Quality: Early timesRoyal Society of LondonTypewriter: the tool that fueled peer review processQuality control criteria of the pastHistory of quality continuedWhat is QualityTraditional Quality control of Scientific Materials: Peer ReviewIs peer review the same as quality?Slide 12The State of QC for PowerPoint on the WebAmazon.com: the first website that allowed to post a book reviewPowerPoint lecturesProgress Report in the SupercourseSpace for written commentsMy dissertation, 20057 expert reviewers from 6 countriesResults and conclusions of this researchImplications for the Supercourse of Science?Where do we go from here for the development of quality control mechanisms in the Supercourse?Supercourse aim:Moving beyond peer reviewSuggested Quality Control Mechanisms for the Supercourse of Science: Multiple MeasuresEditorial Board (in formation)Screening to eliminate poor quality lectureAmazon.com like systemMean Lecture Scores in the SupercoursePersonal Characteristics of Authors (rank, university, citations)Web statisticsKeynote speechesNIH Model and Continuous Quality MonitoringGoogle model for QC: Opportunities for exploration?Why do this research?ConclusionsVision for the Supercourse of ScienceThank you!Themes that Cut Across Disciplines: Quality Control and Research MethodsFaina Linkov, PhDResearch Assistant ProfessorUniversity of Pittsburgh Cancer InstituteThe journey of quality control in the SupercourseThe journey of a thousand miles starts with a single stepOverview•Overview the history of quality control and peer review•Quality control on Amazon.com•Quality control in the Supercourse: QC on PPT lectures is at the core of what we do•Supercourse studies of quality control•Our conclusions and future directionsWhat do scientists have in common?•Quest for discovery•Common research methods, such as statistics•Journal article format•Quality control mechanisms•Peer review: is it the same as quality?History of Quality: Early times•384-322 BCE-Aristotle and the birth of peer review•854-931 “Ethics of the Physician” by Ishap bin Ali Al Rahwi-first documented description of peer review in MedicineRoyal Society of LondonPhilosophical Transactions- Originally edited by Henry Oldenburg1665First peer reviewed journalTypewriter: the tool that fueled peer review process1890’sQuality control criteria of the pastAlbert Einstein, 1905His famous work, Annus Mirabilis, was not peer reviewed!History of quality continued 19th century – birth of quality control discipline1950 - W. Edwards Deming and statistical quality control1970’s- Total Quality Management (TQM)What is Quality“ Doing the right thing right, right away” W. Edwards Deming, 1982 “a measure of goodness that relates to the intended use of a product and the expectations customers have concerning this product” Barkman, 1989Traditional Quality control of Scientific Materials: Peer Review“A peer-reviewed journal is one that has submitted most of its published articles for review by experts who are not part of the editorial staff” Vancouver Group of EditorsIs peer review the same as quality?South Korean cloning scandal, alongside with other examples, demonstrate that peer review is not perfectPeer review congress: finding new ways or fixing the old system?The State of QC for PowerPoint on the Web•Over 7,600,000 PowerPoint files on the Internet give us unique new ways to do QC•Only 3383 lectures tested for QC-Supercourse•We published almost all articles concerning the state QC of PowerPoint on the webAmazon.com: the first website that allowed to post a book reviewPowerPoint lectures•No information on quality control for PowerPoint lectures on-line•There is a growing need to establish quality control for Scientific PPT lectures on Line•Migration of 7 million lectures to the web by 2008•Establishment of “Quality Circles” in the Supercourse: John Last, Vint Cerf, and many othersProgress Report in the SupercourseOver 14000 lecture reviews were obtained in the last 6 yearsLecture review forms utilize Likert like scales, regularly used for the evaluation of educational programs and amazon, ebay, consumers reports..Space for written commentsSome of the URLs have changed since the lecture was developed. It would be helpful if that was corrected, so you could actually see the informationreferred to.Is there any text(s) that you could recommend for epidemiology, i.e., a basic text, an intermediate text, and an advanced text?My dissertation, 2005First evaluation of peer review mechanisms for PPT lectures on-line7 expert reviewers from 6 countriesResults and conclusions of this research•Lectures were rated highly by experts and non-experts•People are primarily rating at the upper tail of the distribution•There is a poor agreement among reviewers (very low Kappa statistic)•The role of personal background in the quality of the peer review is not clear and needs to be further investigated•There is a pressing need for scientific evaluation of PPT lectures on the webImplications for the Supercourse of Science?•There is no consistency about the use of peer review on the web•Utility of peer review strategies for PPT lectures is not clear•New approaches will be developed for the Supercourse of Science. •Shall we allow disciplines to decide the QC techniques which are most applicable for their fields?•There is a need to build a science of QCWhere do we go from here for the development of quality control mechanisms in the Supercourse?Continuous quality improvement (CQI): not a destination, but a journeySupercourse aim:To scientifically develop a system of QC on the web by making multiple measures availablePower of the Internet makes the development of the Science of QC possible.Moving beyond peer review“… We had great difficulty in finding any real hard evidence of the system’s effectiveness, which is disappointing, as peer review is the cornerstone of editorial policies worldwide” Tom JeffersenSuggested Quality Control Mechanisms for the Supercourse of Science: Multiple Measuresa) Expert editorial boardb) Screening to identify non-appropriate lectures, as done in nature and BMJc) Five star system (or Priority Score) similar to that in Amazon.comd) Opinion of experts (professors)e) Personal Characteristics of authors