The Region 4 Stork (R4S) Collaborative Project, Part 5: Site-Specific Customization of Post-Analytical Tools
Accurate identification of newborns with metabolic disease can significantly improve patient outcomes. Conversely, a missed diagnosis can result in significant morbidity and may even result in death. While a false-positive diagnosis does not carry the burden of increased morbidity or mortality, there are social and psychological costs that may generate significant harm. The Region 4 Stork Collaborative was developed to improve detection of true positive cases of metabolic disease and improve accurate diagnosis. The R4S project uses Mayo-developed software that provides postanalytical interpretation of complex metabolic profiles. The R4S project offers physicians worldwide the opportunity to utilize this software to analyze their patients’ test results, and compare them with other locations’ results.
In R4S Collaborative Project Part 5, Piero Rinaldo, M.D., Ph.D., introduces site-specific customization of post-analytical tools.
Dr. Rinaldo is the co-director of the Biochemical Genetics Laboratory, a professor of Laboratory Medicine and Pathology, and a T. Denny Sanford Professor of Pediatrics at Mayo Clinic.
I have a disclosure to make: a provisional patent application related to the content of this presentation has been submitted by Mayo Clinic. The title of the application is “Computer-Based Dynamic Data Analysis.”Thank you for the introduction. This presentation is the fifth segment of a 6 part series describing the products and clinical tools of a newborn screening quality improvement project called Region 4 Stork, or R4S. The title of this presentation is “Site-specific customization of post-analytical tools”.
After completing the overview of the different types of interpretive tools and data entry portals, the topic of this presentation is site-specific customization of the post-analytical tools.
There are 2 types of tool customization in R4S: the first 1is the option to switch from cumulative to own percentile values in the data entry window of the one condition tools. The clinical indications of this action have already been discussed in a previous segment of this series. The main topic of this presentation is the permanent modification in the tool builder functionality of a released tool that is available to all users. This option is driven by the need to fit the diverse analyte panels implemented by individual sites.
The condition models selected for this presentation are 2 inborn errors of metabolism of the proximal urea cycle: ornithine transcarbamylase deficiency, abbreviated as OTC, and carbamylphosphate synthetase deficiency, or CPS. The first of these conditions is a relatively common X-linked disorder; the other is a rare autosomal recessive disorder. The defining clinical manifestation of OTC and CPS is found in the occurrence of acute, potentially life-threatening episodes of hyperammonemia. The biochemical phenotypes of OTC and CPS are not identical but share the finding of low plasma concentration of the amino acid citrulline. Although this abnormality is readily detectable in the amino acid profile and a variety of treatment options are available, OTC and CPS were not included in the recommended uniform screening panel because at the time of the proceedings of the expert panel, between 2004 and 2006, it was felt that there was no screening test with adequate sensitivity and specificity.
However, elevated citrulline is regarded as a reliable marker for several conditions shown here in the Plot by Target Range tool. They are, from left to right, citrullinemia type I, pyruvate carboxylase deficiency, citrullinemia type II, also known as citrin deficiency, argininosuccinic acidemia and maternal citrullinemia type I. 2 of them, citrullinemia type I and argininosuccinic acidemia, were included in the recommended uniform screening panel, citrullinemia type II is one of the secondary targets. For these reasons, the determination of citrulline is included in the vast majority of analyte panels adopted by screening laboratories. The differences between the reference range, the cutoff target range and the respective disease ranges can be better appreciated by an expansion of the Y-axis limited to the boxed area highlighted here.
To date, 121 of the laboratories participating in the R4S collaborative project have selected a high cutoff value for citrulline. Overall, there are more than 500 cases in the true positive database, the vast majority of them is predictably affected with the 2 primary targets, citrullinemia type I, almost 300 cases, and argininosuccinic acidemia, almost 150 cases.
As mentioned in a previous segment of this series, the plot by marker is a tool that allows an unbiased and comprehensive view of disease ranges for a single analyte, in this case citrulline, in all conditions regardless of their clinical significance. On the right side of the plot, it is evident that there are conditions with disease ranges significantly below the reference range.
Going back to the plot by target range, but this time below the reference range, it is possible to recognize the conditions with low citrulline: OTC and CPS, of course, symptomatic female carriers with OTC deficiency, and another rare condition, ornithine aminotransferase deficiency. The total number of OTC and CPS cases combined exceeds 100. On the other hand, only 64 laboratories have selected a low cutoff value. In other words, approximately half of the labs actively monitoring citrulline at the high end apparently would not act on a low value even if the concentration measured in a patient was near zero.
To address this less than ideal situation, the American College of Medical Genetics released in 2012 a new act sheet addressing the finding of decreased citrulline as an actionable marker for the detection of proximal urea cycle disorders in neonatal dried blood spots.
This slide shows the Plot by Condition for OTC/CPS. The display is limited to informative amino acids, and therefore only citrulline is displayed. Although there is clear separation between the reference range, shown as a green shade, and the disease range, shown as the red box, the act sheet correctly underscores a warning that a decreased citrulline concentration alone, in most instances is NOT informative per se and could lead to an excessive number of false positive cases.
The diagnostic utility of low citrulline could be much improved by giving proper consideration to at least seven different ratios. 2 of them, methionine to citrulline and citrulline to phenylalanine, are commonly used as markers of conditions with abnormal levels of methionine and phenylalanine. Citrulline is often used to calculate ratios because this amino acid is the least influenced by total parenteral nutrition and other types of interference. 5 additional ratios are routinely calculated in R4S for the specific purpose to recognize cases with proximal urea cycle defects: these are, from left to right, the alanine to citrulline ratio, the glutamate to citrulline ratio, the glutamine to citrulline ratio, the ornithine to citrulline ratio, and the citrulline to arginine ratio. One of them, the glutamate to citrulline ratio, is the one with the lowest degree of overlap between reference and disease range, only 6.7%. Naturally, the availability of 8 informative markers allows the creation of a post-analytical interpretive tool for OTC and CPS, indeed this tool was one of the Excel-based prototype tools introduced on the R4S website since 2009. The current version is number 16 and was last updated in January 2013.
Just like all other tools, the amino acid concentration values can be uploaded or entered manually in the data entry window and the tool is generated after clicking the Calculate icon. The score comparison plot is highlighted here to call attention to the fact that this particular case is actually the one with the lowest score among the 11 cases, a mixture of retrospective and prospective findings, contributed to R4S by the Minnesota program.
The score section and the score interpretation guidelines of this case are highlighted in this slide. The calculated score, 76, is at the upper end of the “possibly OTC” range (a score between 25 and 80). The all condition tools is also clearly suggestive of the correct diagnosis (diamond placed at the 9th percentile rank among all 44 cases) even if the percentile rank in comparison to the 11 Minnesota cases is listed as zero. So, the tools appear to be working properly and therefore it is legitimate to ask the question: where is the problem?
The problem is clearly outlined in the last sentence of the introductory paragraph of the ACMG act sheet. It reads “Note that some newborn screening laboratories do not report decreased citrulline OR ABNORMAL AMINO ACID RATIOS”. The final comment refers to a different situation because the primary analytes required to calculate a ratio to citrulline are not being measured at all.
For example, the same profile of the true positive case described earlier would not appear in the all conditions tool for a laboratory we will call “Lab 1”. The tool, however, would include a warning that the OTC/CPS tool did not generate a score because one required analyte, ornithine, was not measured by this particular laboratory.
The same outcome takes place for another site, called Lab 2, but the failure to calculate a score is compounded by a second missing analyte, glutamine in addition to ornithine.
Laboratory 3 is missing 3 of the required amino acids, ornithine, glutamine and glutamate, notably the latter is the one required to calculate the most informative marker.
Laboratory 4 does measure ornithine but glutamine, glutamate and alanine are not included.
Finally, Laboratory 5 does not measure ornithine, glutamine, and arginine.
This table summarizes the profiles described in the previous slides: these labs have different amino acid panels, but all of them for one or more omissions are prevented from using the OTC/CPS tool available on the R4S website.
It should be noted that these are not hypothetical situations, they all are true examples selected from the reference percentiles and cutoff profiles of active R4S participating sites. To place this evidence in a more general context, this slide shows the percent of laboratories measuring individual amino acids in addition to phenylalanine, which is used here to normalize percentages. In other words, the number of labs with either calculated percentiles or an active cutoff for phenylalanine is equal to 100%. 7% of these laboratories do not measure methionine. This finding likely reflects the proportion of sites that currently use tandem mass spectrometry to screen exclusively for PKU and MCAD deficiency. A slightly higher proportion, 9%, does not measure citrulline; the difference is likely to reflect the number of labs who have added testing for homocystinuria to a limited panel that already included PKU and MCAD. When arginine is considered, 25% of the laboratories do not measure it. Alanine is close to be evenly split, approximately 50/50, but ornithine, glutamate and glutamine are indeed NOT measured by an overwhelming majority of labs. In summary, 40 to 90% of the participating laboratories do not measure routinely these clinically useful amino acids.
The situation becomes even more concerning when considering the proportion of labs with no evidence, percentiles or cutoff values, of using the 7 ratios that are needed to calculate a score with the OTC/CPS tool, ranging from 48 to 92 percent. Notably, the glutamate to citrulline ratio, the most informative marker for OTC/CPS in neonatal dried blood spots, is NOT measured by 88% of the laboratories participating to the R4S collaborative project. Obviously, there is a lot of room for improvement.
A defining function of the R4S collaborative project is to collect and visualize objective evidence to improve awareness among users of the importance to utilize the full complement of informative markers for every condition. Said that, and to compensate for the pervasive lack of consistency, the R4S tool builder is designed to allow the creation and release of tools with modified marker panels that are site-specific and accessible only to users affiliated with a single site. This process is remarkably simple and is summarized in the 5 steps shown in this slide: create a copy of a general tool, edit markers, verification of score percentiles, editing of interpretation guidelines, and release into production. A trained user can complete this process literally in just a few minutes, using a protocol that is described in details in the following slides.
In the tool builder menu, the selection of the tool to be copied takes place in the list/release window. The list is quickly limited to OTC/CPS by selecting the condition type and the desired condition. There are 3 options available for an active tool: view, archive, and copy. The other icons, edit, release, and delete, are inactive and shown in a lighter color font because they are not applicable to an active tool. Next action is to select COPY.
This action brings up a new dialog window titled “Create Copy for Participant”. The default setting is “Available to All”, to change it just select the drop down menu to display a list of all active participating sites.
For the sake of this presentation, the list has been modified to show only the 5 sites previously identified as Lab 1, 2, 3, 4, and 5.
If lab 1 is selected, the copy of the tool will be specific to this site, and be accessible only to users affiliated with Lab 1.
This selection is finalized by clicking “Save”.
The list now shows the addition of a new tool with a clear indication it is unique to Lab 1. The same action is repeated 4 more times for the other labs, and the list expands accordingly. Notably, all these tools are still invisible to the users of these sites and will remain as such while the necessary modifications takes place before being released into routine production.
When a tool is not yet released, the options that are active in the tool builder are “Edit”, “Release”, “Copy”, and “Delete”. The next step is to select Edit.
In the Edit Marker page, high and low markers are listed in separate tables. The columns of these tables are, from left to right, the name of the marker, the count of results available in the database, and the degree of overlap between reference and disease range. 4 other columns summarize additional attribute of the markers, and will not be discussed in details here. If desired, a complete explanation is accessible in the Documentation menu of the R4S website, section “Presentations”. The last column, with the header “Select” is the one where the inclusion of a marker in the tool is determined.
This slides focuses on the Select column, and shows only the analytes included in the general tool, the one available to all sites. The modification required to make a site-specific tool for Lab 1 is accomplished simply by unclicking the corresponding checkbox, followed by selecting “Save Changes”.
The same process is followed to customize the analyte panels for the other 4 laboratories , again completed by selecting “Save Changes”.
The next step is the verification of the percentile score ranges, and the consequent adjustment of the score interpretation guidelines. As a reminder, the scores corresponding to the 1st, 10th, and 25th percentiles, rounded to the nearest multiple of 5, define the boundaries between the interpretation guidelines: a score lower than the 1st percentile is considered not informative, a score between the 1st and the 10th percentile is informative, but is labeled as “possibly”. Any score greater than the 10th percentile, in this case a value of 80, is increasingly more and more likely to reflect a true positive case. As stated before in this series, scores that exceed the bottom quartile, in other words greater than the 25th percentile of all scores calculated for true positive cases, become increasingly self-evident and should never constitute a diagnostic challenge.
Predictably, the deletion of one or more markers has a direct effect on the scores and the corresponding percentiles, and therefore need to be adjusted in each site-specific tool. In the case of lab 1, the count of cases with all required markers actually increase from 44 to 47 (the total number of cases with at least one value is actually 104), the changes of percentile values are almost negligible. That is not the case for the other tools. Even if the count increases as high as 75 (a 70% increase), the 1st percentile for Lab 2 dropped by more than 50%, from 27 to 12, and it goes even lower, 6, for the other 3 laboratories, lab 3, 4 and 5. Basically, this is an indication of the strength of the site-specific tools in comparison to the general one available to all sites.
The next and final step is to select the icon “Release” for each site-specific tool.
After that action is completed, all tools become available to any user of those sites and are automatically deployed each time the tool runner and/or the all conditions tool are utilized.
To recap the process to create a site specific tool, 5 simple steps are all that is needed: copy tool, edit markers, calculate scores, update guidelines, and release of the tool.
The outcome of this process is shown here. Before customization, lab 1 could not generate a score for this case. After the customized tool was released, the outcome was clearly informative, with a score ranked at the 9th percentile.
The same outcome, and even higher relative ranks because of the reduced percentiles, was observed for each of the other 4 laboratories.
To summarize the clinical utility of tool customization, it is possible to bypass the general tool available to all sites. This flexibility is needed to factor in the great variability of analyte selection adopted by different laboratories. In most cases a site-specific tool is a “scaled down” version that is still sufficient to detect most cases affected with the target condition. However, the possibility of a lower sensitivity should be carefully considered. Finally, tool customization is rapid and easy and could be performed by any user with access to the tool builder. Many sites have taken advantage of the opportunity to create their own customized tools, the only requirement to be met is to be up to date with the submission of all types of data within the scope of R4S: participant profile, reference percentiles, true positive cases, and performance metrics. Because of the very nature of the post-analytical tools, cutoff values are optional as they have become increasingly irrelevant to the interpretation process.
This is the conclusion of part V of the R4S series of Mayo Medical Laboratories Hot Topics. In part VI we will provide an overview of the status of other newborn screening applications within R4S, and describe the most significant improvements and new features of the upcoming version 2.0 of the CLIR software, to be release between late 2013 and first quarter 2014. As a reminder, CLIR stands for Collaborative Laboratory Integrated Reports. We believe CLIR is applicable to a much broader analytical landscape beyond newborn screening, one that covers the entire field of Clinical Biochemical Genetics and potentially many other clinical but also research areas of laboratory medicine and pathology.
Please do not hesitate to contact us if you have any questions or requests related to the content of this presentation. Thank you very much for your attention.