The Region 4 Stork (R4S) Collaborative Project, Part 1: Laboratory Quality Improvement of Newborn Screening
Accurate identification of newborns with metabolic disease can significantly improve patient outcomes. Conversely, a missed diagnosis can result in significant morbidity and may even result in death. While a false-positive diagnosis does not carry the burden of increased morbidity or mortality, there are social and psychological costs that may generate significant harm. The Region 4 Stork Collaborative was developed to improve detection of true positive cases of metabolic disease and improve accurate diagnosis. The R4S project uses Mayo-developed software that provides postanalytical interpretation of complex metabolic profiles. The R4S project offers physicians worldwide the opportunity to utilize this software to analyze their patients’ test results, and compare them with other locations’ results.
In Region 4 Stork (R4S) Collaborative Project Part 1, Piero Rinaldo, M.D., Ph.D., introduces the concept of disease ranges for improved identification of abnormal newborn screening results. Dr. Rinaldo is the co-director of the Biochemical Genetics Laboratory, a professor of Laboratory Medicine and Pathology, and a T. Denny Sanford Professor of Pediatrics at Mayo Clinic.
I have a disclosure to make: a provisional patent application related to some of the content of this series has been submitted by Mayo Clinic. The title of the application is “Computer-Based Dynamic Data Analysis.”Thank you for the introduction. This presentation is the first segment of a 6-part series describing the products and clinical tools of a laboratory quality improvement project called Region 4 Stork, or R4S.
This is an outline of the 6 topics that will be covered in this series. In Part I, we intend to provide background information about the R4S project and also to introduce the concept of incorporating disease ranges in the interpretation of complex metabolic profiles.
R4S started in 2004 as a priority objective of a Regional Genetics Collaborative project funded by the Health Resource and Service Administration (HRSA), which is an agency of the Department of Health and Human Services. After 8 years of funding, in July 2012, the R4S database became part of the Newborn Screening Translational Research Network, an initiative funded by the National Institute of Child Health and Human Development (NICHD).
To properly frame the content of this presentation, it is appropriate to define the goal and other characteristics of this project: simply stated, the GOAL of R4S is laboratory quality improvement of expanded newborn screening. The ENGINE behind this project is CLIR, an acronym that stands for “Collaborative Laboratory Integrated Reports.” CLIR is a multivariate pattern recognition software that can improve significantly the postanalytical interpretation of complex metabolic profiles. Finally, R4S and CLIR are internet based. The URL of the MS/MS Collaborative Project website is shown at the bottom of this slide, www.clir-r4s.org. [January 2020 Update: URL no longer works. ]
This slide shows the home page of the R4S website. Access to this page is password protected and requires prior registration. Said that, new interested users are always welcome and indeed are being added on a daily basis. The home page design of this and of other applications that will be mentioned later in the series is kept constant and is fairly simple to use. From a content perspective, there are 3 major clusters of components. On the top, users can find links to descriptive reports of all the data posted by the site they are affiliated with. In the next section, highlighted in the slide as “peer comparison,” a user can compare the data mentioned above with those of approximately 180 other programs worldwide. Next, there are links to the productivity tools that drive the process of laboratory quality improvement of newborn screening. These tools are the main deliverable of the collaborative project.
The defining characteristics of the R4S project are listed here and will be further explained in the following slides. Briefly, they range from extensive collaboration and data sharing, constructive feedback and peer comparison, and hundreds of productivity and interpretive tools, which are instantly up to date after any data submission and are available on demand via the website.
Although this project started on a relatively small scale with the participation of only 7 Midwest programs in the United States, it has grown significantly since 2004 and at this time 48 US states and territories are now actively involved, together with 131 programs in 49 other countries. To date, there are more than 900 active users with active access to the project website. Beside participation, it is even more important to describe the level of utilization. Since the website went live at the end of 2008, more than 50,000 logins have been recorded, leading to approximately half a million page views. 240 users have attended one of 36 week-long training courses held at the Mayo Clinic in Rochester between 2007 and 2012.
Data sharing, as I mentioned earlier, is without any doubt the foundation of this project. To put it in perspective, in this slide a few descriptive metrics are summarized as they were at the end of January 2013. 77 different conditions are within the scope of the project, more are likely to be added in the future as they are linked to detectable abnormalities of amino acid and acylcarnitine profiles. These profiles can include more than 130 different analytes and calculated ratios. The clinical significance of these markers is based upon the cutoff value that is used to decide if a result is normal or abnormal. The compilation of more than 7,600 cutoff values is another opportunity for peer comparison and constant performance improvement. Yet, the most important metric are the ones listed at the bottom of this slide. By fostering worldwide collaboration, the project database includes more than 14,000 cases affected with one of the 77 target conditions, more than 30,000 reference percentile values, and almost 1 million data points.
So, what is being done with all this data? Well, we make web-based tools, and they are called productivity tools. In this presentation 2 examples of these tools will be discussed, first the plot by condition and then the plot by disease range. In the next presentation, we will illustrate 5 more tools and, finally, in the remaining of the series, we will address what is truly the main deliverable of this entire process, the postanalytical interpretive tools.
To introduce the concept of disease ranges, a good model to use is a disease called very long chain Acyl-CoA dehydrogenase deficiency that, from now on, I will just describe as VLCAD deficiency. This is fitting disease model because more than 600 cases are available. What is VLCAD deficiency? VLCAD deficiency is an autosomal recessive disorder of mitochondrial fatty acid oxidation with an estimated incidence of approximately 1 in 50,000 live births. That figure implies that approximately 1% of the population have in their genes 1 mutant allele of the VLCAD gene, heterozygosity commonly indicated as carrier status that is expected to have no clinical manifestations but could manifest intermittently with a suspicious biochemical phenotype. Like in other fatty acid oxidation disorders, affected patients may not show any evidence of disease for variable periods of time until the occurrence of a first acute episode of metabolic decompensation, triggered by an infection or simply by fasting, an event that could lead to significant morbidity and mortality. Sudden unexpected death of patients with VLCAD deficiency has been reported at any age, from the neonatal period to adult life.
For these reasons, VLCAD deficiency was included as a primary target in the Recommended Uniform Screening Panel (now known as RUSP) that was endorsed by the US Secretary of Health and Human Services in 2010 and since then it represents the required standard of care in the United States. This decision was made in the best interest of affected patients, but it brought up the challenge of properly recognizing at the time of screening a complex and highly variable biochemical phenotype. Because of that, there are performance problems at both ends of the spectrum. There is poor specificity of the screening, meaning a high false-positive rate, and poor sensitivity as well, meaning that the results in affected patients could be interpreted to be normal. These cases are false-negative events and, of course, the undiagnosed patients are exposed to the full spectrum of severe clinical consequences of VLCAD deficiency.
It is therefore extremely important to better understand the biochemical phenotype of VLCAD deficiency.
The biochemical phenotype of VLCAD deficiency is defined primarily by a series of saturated and unsaturated acylcarnitine species with a length of the fatty acid chain between 12 and 16 carbons. The full names of these markers are shown here, but will not be mentioned again in this presentation, and simply be replaced by the abbreviated name shown within parentheses.
There is sufficient consensus in the field to agree that the most informative marker of VLCAD deficiency is C14:1; but, as I mentioned earlier, its concentration in neonatal dried blood spots does not always correlate well with disease status. On one hand it could be within the reference range in affected patients. On the other hand, C14:1 could reach fairly high levels in patients who are just carriers of the disease under stressful environmental conditions or have another, unrelated condition. To improve this situation, a number of ratios have been proposed where the concentration of C14:1 is expressed as a calculated ratio to another marker, either one expected to be normal, like acetylcarnitine, shown as C2, or another species that could also be informative, like C12:1 or C16. Despite the opportunity to rely on 10 different markers, there is no consensus on the proper way to interpret these results, and conflicting views based on expert opinions are abundant in the literature. Today, the debate over using more or less markers, the utility of calculating ratios, and especially what to do to segregate affected patients from carriers continues with persistent intensity.
This is exactly where R4S comes in. By fostering worldwide data sharing it has been possible to create a reference database for each condition. The 1 shown here is a tabular representation of the percentile distribution of all the analytes detected in VLCAD deficiency without any preselection or bias. These are called DISEASE RANGES. It should be self-evident that this collective experience is far more informative than any single site could ever be able to assemble. Minnesota, for example, has diagnosed only 14 cases over 10 years of screening, approximately 2% of the total number of cases included in the R4S database. The disease ranges are the foundation of the first of the tools of the R4S project. The tool is called Plot by Condition.
This table will be progressively populated to show the relationship of each tool with analytes, conditions, participating sites, and so on. Once completely filled, it could represent a useful reference one could use to navigate the website and select the desired tool for a specific task to be completed. As shown here, the Plot by Condition allows the evaluation of all analytes but in a single condition. Access to the tool is achieved by selecting the link of the home page. The selection page is where a user can select a condition type, the condition of interest, and the types of analytes to be displayed. To begin this demonstration, the selection is limited to acylcarnitine species, the group inclusive of the informative marker for VLCAD deficiency as discussed in an earlier slide.
Once “show chart” is selected, the following image is displayed. In this plot, all analyte values are shown on a logarithmic scale, and the disease ranges are expressed not as absolute results, but are first converted to a multiple of the median value from the cumulative reference population of the entire project. Each column represents a single marker, in this plot only acylcarnitines are shown to reflect the selection made in the entry page. The green areas correspond to the reference range, the box plots, either red or gray, correspond to the disease ranges. This insert illustrates the actual percentile values corresponding to each feature of the box plot, the green area spans uniformly from the upper to the lower limit, which are the same percentiles of the box plot. The difference in color of the box plots is actually the intended product of this tool. In R4S, clinical significance is attributed to any analyte where the median of the disease range exceeds either the upper or lower limit of the reference population. In other words, when at least half of the cases have an abnormal result a marker is considered important for the diagnosis of this condition as part of a comprehensive pattern recognition and profile interpretation.
By adding to the display all acylcarnitine ratios stored in the database the plot becomes more complex, but the result is that more informative markers have been added.
In the end, by selecting the option of displaying only informative markers the final result is a clear, evidence-based summary of what could be called the biochemical fingerprint of VLCAD deficiency based on the analysis of neonatal dried blood spots. Not only the informative markers are revealed, but they are also ranked on the basis of the degree of overlap between the reference and disease ranges. This pattern is indeed unique to VLCAD deficiency.
The same process we have summarized, step by step, for VLCAD deficiency can be applied at will to any other condition which is screened for by analysis of acylcarnitine and amino acid profiles in neonatal dried blood spots. This is an example of another condition, methylmalonic acidemia and homocystinuria due to enzymatic defects of cobalamin metabolism known as complementation groups C and D, shown here for convenience as Cbl C,D The plot by condition of the informative species, a combination of acylcarnitine, amino acids, and ratios, is again unique to this condition, showing this time a combination of both high and low informative markers. The final example is a disorder of branched chain amino acid metabolism, commonly called maple syrup urine disease and abbreviated as MSUD. As you can see again, this pattern is unique.
These examples should be sufficient to illustrate the clinical utility of the Plot by Condition, which is the ability to define the complete spectrum of markers with clinical significance for each condition being targeted by a laboratory test. As a reminder, the definition of clinical significance is shown here again. The added value of this plot is also the ability to rank the informative markers on the basis of the degree of overlap between reference and disease ranges.
It is time now to introduce another tool of the R4S project, it is called the Plot by Disease Range. In this tool, only 1 analyte is shown, and for just 1 condition, but it is possible to compare the cumulative disease range of all cases to one based only on the cases contributed by each individual site. Like the previous tool, access is available in the peer-comparison section of the home page.
This tool requires the selection of 1 analyte and 1 condition, and for this demonstration C14:1 and VLCAD deficiency are used again as a model. On the far left, the cumulative disease range and reference range are shown in brighter colors, red and green, respectively.
At the bottom of the plot,the reference range of each site is shown separately together with an additional element, which is the cutoff value adopted by that particular laboratory, shown as a blue triangle pointing upward.
The red boxes show the percentiles of distribution of each site-specific disease range for C14:1 in VLCAD deficiency.
The Plot by Disease range can be customized in different ways. The sorting in descending order of the labs can be based on 1 of 5 percentiles values (here the sorting is based on the 10%ile, the bottom of the box), but also on the cutoff values.
Finally, it is possible to highlight the disease range of your own site, in this case MN, but also the number of cases contributed by each site (in the case of our program, that number is 14).
The clinical utility of the Plot by Disease Range is found in the ability to objectively compare a laboratory own, but often limited, experience to that of other sites. This process could indeed lead to the recognition of a potential risk of missing cases because the cutoff value is set either too high, or too low. To mitigate the concern this plot could be misinterpreted because of excessive inter-laboratory variability, the disease range can be switched to the same format of the plot by condition, a logarithmic scale and conversion of percentile ranges to multiples of the reference median. In the example shown in the insert, the sites are sorted by descending order of the cutoff value. It should be evident that the cutoff values on the left side of the plot are less than ideal, as they are higher than significant portions of the disease ranges from other sites.
This is the conclusion of part I of the R4S series, which included an overview of the project and 2 examples of productivity tools, the Plot by Condition and the Plot by Disease Range. In the next presentation, 5 more types of tools will be presented along with an analysis of their clinical utility for quality improvement of newborn screening by tandem mass spectrometry.
Please contact us via e-mail or by phone if you have any questions or requests. Thank very much for your attention.