| Web: | mayocliniclabs.com |
|---|---|
| Email: | mcl@mayo.edu |
| Telephone: | 800-533-1710 |
| International: | +1 855-379-3115 |
| Values are valid only on day of printing | |
| Web: | mayocliniclabs.com |
|---|---|
| Email: | mcl@mayo.edu |
| Telephone: | 800-533-1710 |
| International: | +1 855-379-3115 |
| Values are valid only on day of printing | |
Region 4 Stork (R4S) Collaborative Project
Hot Topic
Expires: June 24, 2027
Piero Rinaldo, M.D., Ph.D.
Professor of Laboratory Medicine and Pathology,
Mayo Clinic, Rochester, Minnesota
Thank you for the introduction. This presentation is the first segment of a 6-part series describing the products and clinical tools of a laboratory quality improvement project called Region 4 Stork, or R4S.
I have a disclosure to make: a provisional patent application related to some of the content of this series has been submitted by Mayo Clinic. The title of the application is “Computer-Based Dynamic Data Analysis.”
This is an outline of the 6 topics that will be covered in this series. In Part I, we intend to provide background information about the R4S project and also to introduce the concept of incorporating disease ranges in the interpretation of complex metabolic profiles.
R4S started in 2004 as a priority objective of a Regional Genetics Collaborative project funded by the Health Resource and Service Administration (HRSA), which is an agency of the Department of Health and Human Services. After 8 years of funding, in July 2012, the R4S database became part of the Newborn Screening Translational Research Network, an initiative funded by the National Institute of Child Health and Human Development (NICHD).
To properly frame the content of this presentation, it is appropriate to define the goal and other characteristics of this project: simply stated, the GOAL of R4S is laboratory quality improvement of expanded newborn screening. The ENGINE behind this project is CLIR, an acronym that stands for “Collaborative Laboratory Integrated Reports.” CLIR is a multivariate pattern recognition software that can improve significantly the postanalytical interpretation of complex metabolic profiles. Finally, R4S and CLIR are internet based. The URL of the MS/MS Collaborative Project website is shown at the bottom of this slide, clir.mayo.edu [link updated].
This slide shows the home page of the R4S website. Access to this page is password protected and requires prior registration. Said that, new interested users are always welcome and indeed are being added on a daily basis. The home page design of this and of other applications that will be mentioned later in the series is kept constant and is fairly simple to use. From a content perspective, there are 3 major clusters of components. On the top, users can find links to descriptive reports of all the data posted by the site they are affiliated with. In the next section, highlighted in the slide as “peer comparison,” a user can compare the data mentioned above with those of approximately 180 other programs worldwide. Next, there are links to the productivity tools that drive the process of laboratory quality improvement of newborn screening. These tools are the main deliverable of the collaborative project.
The defining characteristics of the R4S project are listed here and will be further explained in the following slides. Briefly, they range from extensive collaboration and data sharing, constructive feedback and peer comparison, and hundreds of productivity and interpretive tools, which are instantly up to date after any data submission and are available on demand via the website.
Although this project started on a relatively small scale with the participation of only 7 Midwest programs in the United States, it has grown significantly since 2004 and at this time 48 US states and territories are now actively involved, together with 131 programs in 49 other countries. To date, there are more than 900 active users with active access to the project website. Beside participation, it is even more important to describe the level of utilization. Since the website went live at the end of 2008, more than 50,000 logins have been recorded, leading to approximately half a million page views. 240 users have attended one of 36 week-long training courses held at the Mayo Clinic in Rochester between 2007 and 2012.
Data sharing, as I mentioned earlier, is without any doubt the foundation of this project. To put it in perspective, in this slide a few descriptive metrics are summarized as they were at the end of January 2013. 77 different conditions are within the scope of the project, more are likely to be added in the future as they are linked to detectable abnormalities of amino acid and acylcarnitine profiles. These profiles can include more than 130 different analytes and calculated ratios. The clinical significance of these markers is based upon the cutoff value that is used to decide if a result is normal or abnormal. The compilation of more than 7,600 cutoff values is another opportunity for peer comparison and constant performance improvement. Yet, the most important metric are the ones listed at the bottom of this slide. By fostering worldwide collaboration, the project database includes more than 14,000 cases affected with one of the 77 target conditions, more than 30,000 reference percentile values, and almost 1 million data points.
So, what is being done with all this data? Well, we make web-based tools, and they are called productivity tools. In this presentation 2 examples of these tools will be discussed, first the plot by condition and then the plot by disease range. In the next presentation, we will illustrate 5 more tools and, finally, in the remaining of the series, we will address what is truly the main deliverable of this entire process, the postanalytical interpretive tools.
To introduce the concept of disease ranges, a good model to use is a disease called very long chain Acyl-CoA dehydrogenase deficiency that, from now on, I will just describe as VLCAD deficiency. This is fitting disease model because more than 600 cases are available. What is VLCAD deficiency? VLCAD deficiency is an autosomal recessive disorder of mitochondrial fatty acid oxidation with an estimated incidence of approximately 1 in 50,000 live births. That figure implies that approximately 1% of the population have in their genes 1 mutant allele of the VLCAD gene, heterozygosity commonly indicated as carrier status that is expected to have no clinical manifestations but could manifest intermittently with a suspicious biochemical phenotype. Like in other fatty acid oxidation disorders, affected patients may not show any evidence of disease for variable periods of time until the occurrence of a first acute episode of metabolic decompensation, triggered by an infection or simply by fasting, an event that could lead to significant morbidity and mortality.
Sudden unexpected death of patients with VLCAD deficiency has been reported at any age, from the neonatal period to adult life. For these reasons, VLCAD deficiency was included as a primary target in the Recommended Uniform Screening Panel (now known as RUSP) that was endorsed by the US Secretary of Health and Human Services in 2010 and since then it represents the required standard of care in the United States. This decision was made in the best interest of affected patients, but it brought up the challenge of properly recognizing at the time of screening a complex and highly variable biochemical phenotype. Because of that, there are performance problems at both ends of the spectrum. There is poor specificity of the screening, meaning a high false-positive rate, and poor sensitivity as well, meaning that the results in affected patients could be interpreted to be normal. These cases are false-negative events and, of course, the undiagnosed patients are exposed to the full spectrum of severe clinical consequences of VLCAD deficiency.
It is therefore extremely important to better understand the biochemical phenotype of VLCAD deficiency.
The biochemical phenotype of VLCAD deficiency is defined primarily by a series of saturated and unsaturated acylcarnitine species with a length of the fatty acid chain between 12 and 16 carbons. The full names of these markers are shown here, but will not be mentioned again in this presentation, and simply be replaced by the abbreviated name shown within parentheses.
There is sufficient consensus in the field to agree that the most informative marker of VLCAD deficiency is C14:1; but, as I mentioned earlier, its concentration in neonatal dried blood spots does not always correlate well with disease status. On one hand it could be within the reference range in affected patients. On the other hand, C14:1 could reach fairly high levels in patients who are just carriers of the disease under stressful environmental conditions or have another, unrelated condition. To improve this situation, a number of ratios have been proposed where the concentration of C14:1 is expressed as a calculated ratio to another marker, either one expected to be normal, like acetylcarnitine, shown as C2, or another species that could also be informative, like C12:1 or C16. Despite the opportunity to rely on 10 different markers, there is no consensus on the proper way to interpret these results, and conflicting views based on expert opinions are abundant in the literature. Today, the debate over using more or less markers, the utility of calculating ratios, and especially what to do to segregate affected patients from carriers continues with persistent intensity.
This is exactly where R4S comes in. By fostering worldwide data sharing it has been possible to create a reference database for each condition. The 1 shown here is a tabular representation of the percentile distribution of all the analytes detected in VLCAD deficiency without any preselection or bias. These are called DISEASE RANGES. It should be self-evident that this collective experience is far more informative than any single site could ever be able to assemble. Minnesota, for example, has diagnosed only 14 cases over 10 years of screening, approximately 2% of the total number of cases included in the R4S database. The disease ranges are the foundation of the first of the tools of the R4S project. The tool is called Plot by Condition.
This table will be progressively populated to show the relationship of each tool with analytes, conditions, participating sites, and so on. Once completely filled, it could represent a useful reference one could use to navigate the website and select the desired tool for a specific task to be completed. As shown here, the Plot by Condition allows the evaluation of all analytes but in a single condition. Access to the tool is achieved by selecting the link of the home page. The selection page is where a user can select a condition type, the condition of interest, and the types of analytes to be displayed. To begin this demonstration, the selection is limited to acylcarnitine species, the group inclusive of the informative marker for VLCAD deficiency as discussed in an earlier slide.
Once “show chart” is selected, the following image is displayed. In this plot, all analyte values are shown on a logarithmic scale, and the disease ranges are expressed not as absolute results, but are first converted to a multiple of the median value from the cumulative reference population of the entire project. Each column represents a single marker, in this plot only acylcarnitines are shown to reflect the selection made in the entry page. The green areas correspond to the reference range, the box plots, either red or gray, correspond to the disease ranges. This insert illustrates the actual percentile values corresponding to each feature of the box plot, the green area spans uniformly from the upper to the lower limit, which are the same percentiles of the box plot. The difference in color of the box plots is actually the intended product of this tool. In R4S, clinical significance is attributed to any analyte where the median of the disease range exceeds either the upper or lower limit of the reference population. In other words, when at least half of the cases have an abnormal result a marker is considered important for the diagnosis of this condition as part of a comprehensive pattern recognition and profile interpretation.
By adding to the display all acylcarnitine ratios stored in the database the plot becomes more complex, but the result is that more informative markers have been added.
In the end, by selecting the option of displaying only informative markers the final result is a clear, evidence-based summary of what could be called the biochemical fingerprint of VLCAD deficiency based on the analysis of neonatal dried blood spots. Not only the informative markers are revealed, but they are also ranked on the basis of the degree of overlap between the reference and disease ranges. This pattern is indeed unique to VLCAD deficiency.
The same process we have summarized, step by step, for VLCAD deficiency can be applied at will to any other condition which is screened for by analysis of acylcarnitine and amino acid profiles in neonatal dried blood spots. This is an example of another condition, methylmalonic acidemia and homocystinuria due to enzymatic defects of cobalamin metabolism known as complementation groups C and D, shown here for convenience as Cbl C,D The plot by condition of the informative species, a combination of acylcarnitine, amino acids, and ratios, is again unique to this condition, showing this time a combination of both high and low informative markers. The final example is a disorder of branched chain amino acid metabolism, commonly called maple syrup urine disease and abbreviated as MSUD. As you can see again, this pattern is unique.
These examples should be sufficient to illustrate the clinical utility of the Plot by Condition, which is the ability to define the complete spectrum of markers with clinical significance for each condition being targeted by a laboratory test. As a reminder, the definition of clinical significance is shown here again. The added value of this plot is also the ability to rank the informative markers on the basis of the degree of overlap between reference and disease ranges.
It is time now to introduce another tool of the R4S project, it is called the Plot by Disease Range. In this tool, only 1 analyte is shown, and for just 1 condition, but it is possible to compare the cumulative disease range of all cases to one based only on the cases contributed by each individual site. Like the previous tool, access is available in the peer-comparison section of the home page.
This tool requires the selection of 1 analyte and 1 condition, and for this demonstration C14:1 and VLCAD deficiency are used again as a model. On the far left, the cumulative disease range and reference range are shown in brighter colors, red and green, respectively.
At the bottom of the plot, the reference range of each site is shown separately together with an additional element, which is the cutoff value adopted by that particular laboratory, shown as a blue triangle pointing upward.
The red boxes show the percentiles of distribution of each site-specific disease range for C14:1 in VLCAD deficiency.
The Plot by Disease range can be customized in different ways. The sorting in descending order of the labs can be based on 1 of 5 percentiles values (here the sorting is based on the 10%ile, the bottom of the box), but also on the cutoff values.
Finally, it is possible to highlight the disease range of your own site, in this case MN, but also the number of cases contributed by each site (in the case of our program, that number is 14).
The clinical utility of the Plot by Disease Range is found in the ability to objectively compare a laboratory own, but often limited, experience to that of other sites. This process could indeed lead to the recognition of a potential risk of missing cases because the cutoff value is set either too high, or too low. To mitigate the concern this plot could be misinterpreted because of excessive inter-laboratory variability, the disease range can be switched to the same format of the plot by condition, a logarithmic scale and conversion of percentile ranges to multiples of the reference median. In the example shown in the insert, the sites are sorted by descending order of the cutoff value. It should be evident that the cutoff values on the left side of the plot are less than ideal, as they are higher than significant portions of the disease ranges from other sites.
This is the conclusion of part I of the R4S series, which included an overview of the project and 2 examples of productivity tools, the Plot by Condition and the Plot by Disease Range. In the next presentation, 5 more types of tools will be presented along with an analysis of their clinical utility for quality improvement of newborn screening by tandem mass spectrometry.
Please contact us via e-mail or by phone if you have any questions or requests. Thank very much for your attention.
Piero Rinaldo, M.D., Ph.D.
Professor of Laboratory Medicine and Pathology,
Mayo Clinic, Rochester, Minnesota
Thank you for the introduction. This presentation is the second segment of a 6-part series describing the products and clinical tools of a newborn screening quality improvement project called Region 4 Stork, or R4S.
I have a disclosure to make: a provisional patent application related to some of the content of this series has been submitted by Mayo Clinic. The title of the application is “Computer-Based Dynamic Data Analysis.”
To briefly recap the first presentation, 2 types of tools were introduced: the plot by condition, shown here on the left side, where all markers for 1 condition can be evaluated to establish clinical significance, and the plot by disease range, where a single analyte for a single condition is presented as a separate disease range for each individual site that has contributed at least 1 case with the chosen condition.
The summary table that is shown in this slide compares the content and purpose of the 2 tools already discussed with 5 more types of first-generation tools:
They are the plot by marker, the plot by target range, the score card report, the interactive scatter plot and finally the analyte comparison tool. The objective of this presentation is to illustrate how the tools could be deployed to describe a laboratory own status and performance in a context of objective and evidence-based peer comparison, 1 that is always up-to-date and available on demand.
The first tool is the plot by marker. This tool could be described as the opposite combination of the plot by condition, because it shows the disease ranges of just 1 marker, but in all conditions regardless of the achievement of clinical significance. Just like all other tools, access to the plot by marker is provided by a link available on the web home page.
In the selection window, we continue to use as a model C14:1, the primary marker of VLCAD deficiency. The analyte type is, of course, acylcarnitines.
C14:1 is selected from the analyte drop down menu as shown in this animation. As a final step, 1 or all condition types can be selected. It is recommended to select at first all 3 types of conditions (amino acid, fatty acid, and organic acid disorders) to cover the full spectrum of conditions.
After clicking “show chart”, the following image is displayed. Most elements are kept consistent from tool to tool, like the representation of the cumulative reference range of C14:1, shown both as a green shade as well as a box plot of the 5 percentile values that were introduced in the previous presentation.
Disease ranges are color coded, bright red represents conditions where C14:1 is informative according to the rule that the median value, shown as a white circle, is positi1d above the upper limit of the reference range, shown as a green shade. The insert shows an expanded view of the C14:1 informative disease ranges: VLCAD deficiency and VLCAD carriers, glutaric academia type II (abbreviated as GA-2), and 2 conditions which are shown together because their biochemical phenotypes are essentially identical. They are long-chain 3-hydroxy acyl-CoA dehydrogenase deficiency, LCHAD, and trifunctional protein deficiency, or TFP.
All other conditions are shown as gray boxes, the respective medians are well within the reference limits.
What is a clinical utility of the plot by marker tool? The plot by marker is very useful to recognize the extent of differential diagnosis that is required to properly interpret an informative result for a given analyte. It is important to emphasize here that testing by mass spectrometry for the recommended newborn screening uniform panel involves a large number of markers and calculated ratios, more than 100 overall. The vast majority of them require a differential diagnosis that could span from just 2 to 13 conditions, with an average of 5. In other words, when a result is abnormally high or low, a direct relationship to just 1 possible diagnosis should be considered the exception rather than the rule. In our experience, proper consideration of all possible outcomes is indeed essential to properly select cutoff values.
Selection of a cutoff value is best achieved by using the next productivity tool, the plot by target range. This tool is different from the previous 1 in 2 aspects: it shows only clinically significant disease ranges and also adds between reference and disease ranges a new element, the cutoff disease range. The link to access the plot by target range, like all other tools, is found in the website homepage.
A different analyte has been selected as the model to demonstrate the purpose of this tool. Xle is an abbreviation commonly used to indicate the mixture of 2 branch chain amino acids that have the same molecular weight (isoleucine and leucine). In a testing mode that does not involve front-end chromatographic separation, isobaric compounds like these 2 and also other minor components are measured together. Elevated concentrations of these 2 markers combined are informative for just 1 condition, maple syrup urine disease, shown in the slide as MSUD. From the left side, the 3 sections of this plot correspond to the reference, cutoff, and disease range of isoleucine/leucine, respectively. The cumulative percentiles are based on the data contributed by 115 laboratories worldwide, equivalent to several millions of newborns; the cutoff range is calculated from 127 independent values; the total number of true positive cases is 283. These cases have been uploaded by 87 different programs, each contributing between just 1 to 27 cases, with a median of 2 per site. The ordinate scale (the Y-axis) of all R4S plots can be adjusted as needed. For example, changing the scale from the default level of greater than 3,000 nmol/mL to approximately 800 is done by a click of the mouse at the desired level.
Once the scale has been reduced, it is possible to better appreciate these ranges, but also to introduce the concept of cutoff target range. For a disease where clinical significance is found above the reference population, like in this case, the lower limit corresponds to the 99th percentile of the reference range. The upper limit is equal to the 5th percentile of the disease range. This choice, rather than either the lowest value observed for a given condition or the 1st percentile of the disease range, is necessary because virtually in every condition there are highly unusual cases with normal or even lower than normal results. These cases would be missed by any cutoff value for isoleucine/leucine a lab would choose, and any attempt to identify then would trigger unmanageable numbers of false- positive cases. The interval between the 2 levels is what is called the CUTOFF TARGET RANGE. As I mentioned earlier, there is 1 important and interesting observation to be made here: the MEDIAN value of the 127 cutoffs used in screening practice worldwide is ABOVE the target range. This situation is likely to cause multiple false-negative events that could be avoided by just selecting a cutoff values that lies within the target range. There is another informative marker of MSUD, the branched chain amino acid valine. Could valine compensate for the poor specificity of isoleucine/leucine?
Indeed that is not the case. The same plot for valine shows an even greater overlap between reference and disease ranges, and also a target range that had to be adjusted to compensate for this situation. The median of the cutoff range is even higher than in the case of isoleucine/leucine revealing a trend of selecting cutoff values on the sole basis of the reference range, therefore penetrating very deep in the MSUD disease range with a degree of overlap of almost 70%.
The concern for the selection of cutoff values for these markers is unfortunately shared by many others. Another compelling example is propionylcarnitine, an acylcarnitine species commonly abbreviated as C3. Like in the case of MSUD, we begin with the plot by disease range for a single condition, propionic academia. Just like before, the median of the cutoff range is higher than the target range. This situation is actually more concerning once the question is asked if C3 is informative for other conditions beside propionic acidemia.
It is a fact that high concentrations of C3 could be found in as many as 8 other conditions, shown here only by their abbreviated names. More importantly, the corresponding disease ranges are lower than the one found in propionic acidemia. Clearly, the degree of overlap between all disease ranges and the cutoff range becomes so pervasive that it is not rare to see cutoff values set at a level higher than the median of some disease ranges. Basically, this means that there is a greater than 50% chance that an affected case could be missed by newborn screening, causing false-negative events that would have been possible to avoid by a more in-depth clinical validation of cutoff values.
It should be mentioned that C3 is also informative BELOW the reference range. This plot shows that there are at least 5 conditions where a low concentration of C3 is actually very informative. While 120 laboratories have implemented a high cutoff value for C3, to date only 50 of them have also adopted a second cutoff at the low level. In conclusion, C3 in indeed the marker where the full extent of differential diagnosis should include 13 different conditions.
For a summary of the clinical utility of the plot by target range, consider reading this 2011 publication entitled “Clinical validation of cutoff target ranges in newborn screening of metabolic disorders by tandem mass spectrometry: a worldwide collaborative project”. In this paper, the disease ranges were based on less than 11,000 true-positive cases, equivalent to the status of the project at the end of 2010. While the size of this population study is unprecedented, it should already be considered out of date already. Just 2 years later, at the end of 2012, the count had grown by a third to more than 14,000. Since 2009, an average of 5 new cases are uploaded to the database every single day.
This constant growth is exactly the reason why the next productivity tool has been created. The score card provides a dynamic, instead of a static, report of the cutoff target ranges, one that is kept constantly up to date. The score card summarizes the data of all significant analytes and ratios for all conditions. Access is from a link on the homepage, like all other tools.
This slide shows the selection window of the score card. Please note the additional link shown at the top left corner of the blue panel. By clicking “How to use this tool”, a user can access a detailed procedure that explains the content of this tool and how to use it properly.
Access to this training material is provided in 3 ways: from the menu bar under the heading Documentation, from the home page, and as shown already from the selection window of every tool.
The list of available training documents is shown in this slide. There is a document available for each of the tools presented so far: the plot by condition, the plot by marker, the plot by target range, and of course, the score card.
Going back to the score card selection window, the type of analyte is the only choice to be made. Once one has been chosen, like in this example, the score card for that type of marker is displayed.
The score card summarizes in a tabular rather than a graphic way all the data shown as box plots in the previous tools. Each section includes a current count of contributions, either sites or cases, and cumulative percentile values: reference ranges by analyte on the left side, the high and low disease ranges on the right side. This section is sorted by individual conditions and also includes 3 additional percentile values: the 5th, the 25th, and the 75th percentile. In the central portion of the score card, there is a summary of high and low cutoff ranges, and finally the high and low cutoff target ranges. If needed, a detailed explanation of the color coding is included in the training material available on the website. Briefly, when the background color of the target range is either yellow or purple, it indicates that there is a complete separation between the reference range and the disease range of the chosen target condition. When the background color is black, it means a manual adjustment was necessary because the 2 ranges do overlap. One of the analytes discussed previously, C3, is shown in this slide. As a reminder, all these ranges are updated instantly after any new submission to the website.
This is another portion of the acylcarnitine score card tool that shows the data for the other analyte we have discussed previously, C14:1. The disease ranges on the right side are condition specific, showing, for example, the differences between VLCAD deficiency and VLCAD carriers.
If amino acids were chosen as analyte type, the following page would appear, showing at the top the branched-chain amino acids that have been used as model to illustrate the plot by target range tool.
To appreciate the magnitude of the information available in this tool, this slide shows the full sets of ranges available for the 4 analyte types: amino acids, amino acid ratios, acylcarnitines and, on the far right, acylcarnitine ratios. As a reminder, the score cards are always up-to-date and, of course, are available on demand.
The next tool presented in this series is called the scatter plot. This plot is very interactive in nature, as it provides a powerful way to investigate the distribution of true-positive results not as cumulative ranges but as individual cases. The plot shows the actual results of any combination of 2 markers, and as many conditions as desired.
This is the entry page of the scatter plot. The model for the demonstration of how to use it is provided again by the 2 branched-chain amino acids, isoleucine/leucine and valine.
A user needs to select the analyte type, the specific analyte for the X and Y axes, the condition type, and finally the condition of interest. To select more than 1 condition, hold the CTRL key while highlighting with the mouse all the conditions to be displayed. For this demonstration, the median value of the 2 cutoff ranges is also included, 278 for isoleucine/leucine and 250 for valine. The scatter plot offers the option to display or hide all other true-positive cases. In this example, it has been deactivated by unclicking the box at the bottom of the window.
When “show chart” is selected, the following image is displayed. The pair of results for each case with MSUD is shown as a blue dot. The scale is automatically set to 20% more of the highest value on record, so the reference limits are compressed at the bottom left corner of the plot. This is easily changed by clicking the mouse once within the plot to reformat the 2 axes. To continue this presentation, both axes are reset to a value of approximately 1,000.
The vertical and horizontal lines of the green area corresponds to the 1% and 99% percentiles of the reference population, the dotted red lines represent the cutoff values entered in the selection page. As a reminder, they are shown on the left side of the slide together with the percentile of the cutoff range they represent, in this case the 50th, which is the median, and the count of values included in the range (128 and 115, respectively). The red lines divide the plot in 4 quadrants. The top right quadrant shows all cases with results exceeding both cutoff values. At the opposite end of the spectrum, the bottom left quadrant shows how many cases would have been missed using the same parameters. Indeed, some of them are documented false-negative events added to the R4S database after clinical ascertainment. The other 2 quadrants show what we may call the mixed results, 1 positive and 1 negative. Clearly, many cases with MSUD had a valine result below the median cutoff value, information that could help in the interpretation of challenging cases.
What happens when different cutoff values are selected? If, for example, we plot the respective 1stpercentiles of the cutoff ranges (176 for isoleucine/leucine and 143 for valine) the detection of true positive cases becomes almost perfect, but the overlap with the reference population, the green area, would cause an unacceptably high number of false- positive cases.
At the opposite end of the cutoff ranges, the cutoff values corresponding to the 99% percentiles, which are 492 for isoleucine/leucine and 450 for valine, a majority of affected cases would be missed. This is far from being a hypothetical situation because these are the actual values used by 1 laboratory in its routine screening practice.
In summary, the scatter plot could be viewed as a testing environment where a laboratory could verify its choices of cutoff values against the cumulative experience worldwide. As an example, we consider a laboratory with cutoff values for isoleucine/leucine and valine of 315 and 300, respectively.
The scatter plot clearly shows that at those levels the risk of false-negative events is fairly high considering the high number of cases with 1 or both results below the chosen cutoffs.
The scatter plot is particularly useful in the prospective evaluation of challenging cases. For example, the same laboratory has to provide an interpretation for a newborn with the following results: isoleucine/leucine 300 nmol/mL and valine 283. These values can be entered in the selection page to generate the next image.
The red diamond and the arrow show where this case is located. Although the results are below the cutoff values, there are several true-positive cases with similar and even lower results. This could be a very challenging situation. Fortunately, a review of the demographic information of this patient reveals that at the time of sample collection the newborn was on total parenteral nutrition, or TPN.
When TPN is added to MSUD as the conditions to be displayed, the image is modified as shown here. The orange dots indicate TPN cases, and indeed, this case is located very close to the majority of them. However, the overlap between MSUD and TPN cases remains unresolved.
When the type of alimentation is TPN, the concentrations of additional amino acids could be elevated, particularly phenylalanine, shown as Phe and methionine, shown as Met. The values in this case, like the branched-chain amino acids, can be described as borderline in reference to the median cutoff values: 150 for phenylalanine and 60 for methionine.
However, plotting the phenylalanine and methionine values in MSUD and TPN cases shows an improved separation between the cluster of MSUD cases and the case under evaluation, shown again and a red diamond and by the blue arrow.
As a final step, the introduction of ratios can provide the conclusive evidence to resolve this case. Shown here are 2 ratios, isoleucine-leucine/phenylalanine and methionine/ isoleucine-leucine, the median cutoff values of the R4S project, and the calculated values in this patient, 2.07 and 0.23, respectively.
Considering the position of this case and of the cluster of TPN cases, the separation between MSUD and TPN cases is virtually complete, and this case could be closed with confidence as a nutritional artifact that does not require a referral to follow-up and the collection of a repeat specimen and/or additional laboratory investigations.
The final plot of this presentation is the analyte comparison tool. It allows the single simultaneous evaluation of the basic elements of any analyte in comparison to the collective experience of all other laboratories.
Once again, the model analyte is isoleucine-leucine. The tool shows the range of the cumulative reference percentiles, the anonymized reference percentiles of each individual contributing site, the disease range of all conditions where the chosen analyte is informative, in this case just 1, MSUD, the distribution of individual cutoff values shown next to the disease range to easily appreciate the degree of overlap, the cutoff target range proposed by the R4S project, and the option to highlight the reference percentiles and cutoff value of your own laboratory. The size of the blue or purple diamond is proportional to the number of laboratories that have chosen the same cutoff value. Like all other tools, the analyte comparison tool is kept instantly up to date after any new data submission and is available on demand.
This is the conclusion of the second part of this series. In part 3 we will begin to illustrate how the evidence-based approach and concepts of the first-generation tools have been utilized to create the second generation of tools, which have been called the postanalytical interpretive tools of the R4S project.
Thank you for your attention. Please do not hesitate to contact us if you have any questions or requests related to the content of this presentation.
Piero Rinaldo, M.D., Ph.D.
Professor of Laboratory Medicine and Pathology,
Mayo Clinic, Rochester, Minnesota
Thank you for the introduction. This presentation is the third segment of a 6-part series describing the products and clinical tools of a newborn screening quality improvement project called Region 4 Stork, or R4S. The title of this presentation is “post-analytical interpretive tools.”
I have a disclosure to make: a provisional patent application related to some of the content of this presentation has been submitted by Mayo Clinic. The title of the application is “Computer-Based Dynamic Data Analysis.”
This presentation focuses on the second generation of R4S tools. It will take this and 2 more segments to complete the overview of these tools and of their clinical utility.
We begin with a summary of the productivity tools that have been described previously. What they all have in common is the goal to collect and analyze the data needed to establish clinically validated cutoff target ranges for the markers measurable by tandem mass spectrometry in neonatal dried blood spots.
This publication summarized the status of the R4S project at the end of 2010. At that time, a total of 5,341 cutoff values had been uploaded to the R4S database and in the paper were sorted according to their standing in comparison to the corresponding target range. The results of this analysis were as follows: 2,269 (or 42%) were set properly within the limits of the respective target range; 788 (or 15%) were actually positioned unnecessarily too close to the reference range, and were likely to cause recurrent false positive outcomes. The remaining 2,284 (or 43%) had the opposite and more concerning problem: these cutoff values were overlapping with the disease range of one or more conditions and therefore were intrinsically at risk of missing affected cases, resulting in false negative events. This observation was compounded by the fact that more than 40% of these cutoff values with an intrinsic risk of low sensitivity were applied to 37 analytes and ratios that had no overlap between reference and disease ranges. Despite this compelling evidence, overall, there was only limited response to the publication of this work and to date only a small fraction of the inadequate cutoff values had been corrected by the laboratories participating in the R4S project.
For this reason, it became a priority to explore ways to rely more on the power of pattern recognition and profile interpretation of complex metabolic profiles rather than depending on cutoff values, even when their clinical validation had been accomplished. In this presentation, the content of the second publication of the collaborative project will be illustrated to show how the goal of interpreting newborn screening results without analyte cutoff values could be achieved.
This new table shows the basic characteristics of the post-analytical tools. The 2 columns on the right side have been obsoleted, and since 2011 it is fully possible to interpret newborn screening profiles without using cutoff values and related target ranges.
In their place, the summary table of the post-analytical tools includes information about the applicability of each tool to either one or two multiple cases, and a synopsis of the standardized interpretation guidelines included in each tool.
Before discussing specific tools, it is important to give a brief overview of how these tools are actually created. This process is called the Tool Builder.
What is the Tool builder? The tool builder is the foundation of a multivariate pattern recognition software that creates either completely new or modified versions of the different types of post-analytical interpretive tools. When a user is given access to this function, he or she can make a customized site-specific tool following a sequence of 10 relatively simple steps shown on the right side of the slide, a process that after proper training can take as little as 5 minutes to complete. The post-analytical tools are instruments of clinical utility and work best when used to provide an answer to one of three types of questions: a yes or no situation (does a patient have or not a specific condition?); a differential diagnosis between two conditions with similar biochemical phenotype (for example, VLCAD deficiency and VLCAD carrier status); and to answer simultaneously the yes or no question for more than two and up to hundreds of conditions if analyte disease ranges have been properly established to allow the release of a tool targeting each of those conditions. A defining characteristic of the post-analytical tools is the evolution of clinical validation from the conventional static process, one usually performed early during test development, to a constantly evolving, dynamic refinement of the disease ranges that continues to improve throughout the entire test life cycle process.
In the following slides, the basic concepts of the post-analytical tools will be discussed. First, we will compare the benefits of a parallel rather than sequential evaluation of potentially informative markers. As an example, let’s use a hypothetical condition were the biochemical phenotype is defined by two informative markers, A and B, and two related ratios (the A to C ratio and the B to C ratio). In a traditional sequential model, a decision point is set for each marker by a fixed cutoff value, shown here as generic letter (W, X, and Y). Assuming these cutoffs are above the respective reference range (in other words they are “high” markers), if the patient’s value is below the cutoff at any of these decision points the outcome would be a negative result. If all four markers are abnormal, only then the test will be interpreted as positive. The insert shows an actual sequential algorithm of comparable complexity, one that applies to the combined interpretation of 5 steroids measured as part of a newborn screening 2nd tier tests for congenital adrenal hyperplasia. The most useful marker from a diagnostic perspective is a calculated ratio, the sum of two precursors, 17OH Progesterone and Androstenedione, divided by the end product of that biosynthetic pathway, cortisol. Based on a statistical analysis, the cutoff value for this ratio was set at a value of 2.5. It can be argued that a marginal difference of just 0.1 in the calculated ratio is unlikely to reflect a true separation between true positives and true negatives.
Sequential algorithms are also affected by yet another source of variability, the making of rules according to the “AND/OR” model. In a pure “AND” model, each element of the algorithm must meet the definition of abnormal, and a single exception is sufficient to trigger a negative outcome, regardless of the magnitude of the other results. Because of its intrinsic inflexibility, algorithms may include the concept of “OR”. For example, if either analyte A OR analyte B exceeds the respective threshold, that is sufficient evidence to overrule the other result and move on to the next decision point, if any. In an entirely “OR” model, like the one shown on the far left, as long as one of the potentially informative markers exceeds the respective cutoff value the cumulative interpretation is that the test is positive. Obviously, there could be countless combinations of these rules that rapidly become almost impossible to memorize and are challenging to keep updated in a laboratory information system. For this reason, a parallel algorithm is intuitively a better process.
In a parallel model, all analytes are considered simultaneously. The entire profile of primary data is processed by a condition-specific post-analytical interpretive tool. The tool formulates a cumulative score based on the pattern of all analytes expected to be informative and the score is expressed as the percentile rank in comparison to all known cases with the target condition. The score and its relative ranking are applied to standardized guidelines that allow to interpret the overall profile to be either negative or positive for that condition.
Another critical difference between a cutoff-based protocol and the post-analytical tool is the progressive scoring of individual results on the basis of two integrated dimensions: the degree of overlap between reference and disease range, a concept that was illustrated in a initial presentation of this series, and the degree of penetration within the portion of the disease range that does not overlap with the reference range. To elucidate this concept, we will use again VLCAD deficiency as a model. The column showing the actual results of this patient is highlighted on the right side. The analyte shown in the top row of this table is the acylcarnitine species C12:1, and the measured value is 0.50 nmol/mL. On the left side, the table shows that the 99th percentile of the reference population is equal to 0.27 nmol/mL. As the degree of overlap between reference and disease range is equal to 34%, a result of 0.5 will not generate a score until it exceeds the 30th percentile of the disease range, which is 0.25 nmol/mL. From that point on, the crossing of each percentile thresholds that is lower than the patient’s value of 0.5 does generate a score. C12:1 contributes to the cumulative score of this case by exceeding the 40th, 50th, 60th, and 70th percentiles of the disease range. The same process takes place for every informative marker: C14, with a degree of overlap of only 4%, generates a score starting from the 5th percentile of the disease range up to the 70th percentile; C14:1, with no overlap, generates a score immediately after exceeding the 1st percentile of the disease range and continues to do so all the way up to the 80th percentile. This action is repeated for all remaining analytes and ratios simultaneously, and that is why this process is called a parallel algorithm, where there are no dependencies between one analyte and another (the “and/or” rules). Different scoring models are available, but their discussion in details is beyond the scope of this presentation.
The scoring model of this particular tool is the increasing type, the one that is most commonly used. In this model the first crossed percentile beyond the overlap with the reference range counts for a score of 1, the second for a score of 2, with increments of 1 up to a maximum of 12. In this case, the contribution of C12:1 to the cumulative score is 10, the sum of 1+2+3+4. All individual scores are added together, ready for the final modification based on correction factors.
As the final step, proportional correction factors are applied to convert the preliminary scores to a final value. These factors are weighted to reflect the degree of overlap between reference and disease ranges, as best documented by the plot by condition, and are unique to the condition under consideration. The final score is equal to the sum of all individual scores multiplied by the respective correction factor.
The post-analytical tools also incorporate a series of rules which have been included for different purposes: Differentiator rules prevent generating a score driven by significant abnormalities of non-informative markers which are used to calculate ratios. This may happen in cases actually affected with a different condition; Outlier rules prevent generating a score driven by significant abnormalities of non-informative markers per se, not used to calculate ratios. This is a requirement unique to the all conditions tool that will be presented in the next segment; Filters prevent skewing the distribution of scores and percentile ranking on the basis of absurd values, errors in data entry, but also unavoidable true negative cases. Their need is driven by the web-based, always up-to-date nature of the tools, where any new information uploaded to the database is incorporated instantly in the corresponding tool.
After this brief overview of the basic concepts behind the post-analytical tools, it is time to introduce the first one: the one condition tool. Access to the secondary page with the links to different groups of tools is directly from the home page, like all others.
If a user selects the one condition tool link, the following page is displayed, showing at the top the link to the next window where the tools are listed.
One condition tools are split in groups according to the condition types. In the MS/MS application there are 3 groups: Amino acid disorders, left, fatty acid oxidation disorders, center and organic acid disorders, right. Each row is an active link that follows a constant nomenclature and format. To illustrate these features, we will use the tools for 3 conditions we have used as a model previously: MSUD, Cbl C,D, and VLCAD deficiency.
The tool name consists of four elements: the abbreviated name of the condition, the tool version (in this case, the current version of the VLCAD tool is #15), the date it was released (in a year-month-day format), and the type of tool. All other tools are named following the same format.
There are 6 components of the one condition tool and they are listed here.
The first 1 is the data entry window, shown here for the 3 model conditions. In the lower section, the data entry fields are empty and are arranged in 3 groups: low markers, only shown in the Cbl C,D tool on the right side, differentiators, and high markers. These analytes correspond to the biochemical phenotype established by the respective plot by condition.
Data can be entered either manually or, preferably, by uploading a comma separated value (abbreviated as csv) file. The score is calculated by clicking the calculate button at the lower end of the window.
The first element to be displayed at the top of the report page is the tool banner. In addition to the information included in the name of the tool, the banner also shows the date and time the tool was used, the date of the last modification, the site affiliation and the name of the user who calculated the score.
The next element of the tool is a tabular summary of the reference and disease ranges of the informative analytes. From left to right, the columns show the analytes and ratios, sorted in the 3 groups mentioned earlier to illustrate the data entry window (low markers, differentiators, and high markers).
The limit of the reference ranges on the side closest to the disease range.
The degree of overlap between the reference range and the condition-specific disease range, sorted in decreasing order of clinical significance (from least to most).
A selection of disease range percentiles superimposed to a shaded background when overlap is present.
And the actual patient results. Ratios are calculated by the tool and need not to be uploaded in the data entry window.
The next section is the plot by condition, showing for each analyte the patient result, the percentiles of the disease range, and the 1st to 99th range of reference percentiles.
While the top and middle sections of the tool report are descriptive in nature, the bottom section, called Score Report, is where the calculated score is presented, compared, and interpreted.
There are 3 components of the score report: In the top left section, a table summarizes the case score calculated using disease ranges derived from the entire set of available cases (shown as “all” on the left side), and the scores calculated using only cases from a single country, and just from the single site of the user. The score are also shown as the percentile rank in comparison to the true positive cases of each set. The actual number of cases is shown at the bottom. In addition, this section also includes a link called View Calculations.
By clicking View Calculations is it possible to review the contribution of every analyte and ratio to the calculated score. This is a critical feature of the post-analytical tool to document that scoring is a transparent process based only on objective evidence that is fully accessible to users.
The second element of the score report is the comparison plot. In this figure, individual scores from the true positive database are shown as individual black lines. The same distinction between entire project, single country, and single site is reproduced here. In each column, the calculated score of the case under consideration is shown as a red diamond.
The final element of the score report of the one condition tool is a textual description of the interpretation guidelines. More than the actual value, the interpretation of the cumulative score is based on its percentile rank when compared to the population of true positive cases. A score lower than the 1st percentile of all VLCAD scores (in this case a value of 30) is considered to be NOT informative for VLCAD deficiency; a score between the 1st and 10th percentile is considered to be possibly informative; a score between the 10th and the 25th percentile is likely to be VLCAD deficiency and finally a score above the bottom quartile, in other words greater than the 25th percentile of all scores, is considered very likely to be consistent with a diagnosis of VLCAD deficiency. Although these thresholds are used consistently in all the tools available to users from every site, the so called “General tools”, the selection of interpretation guidelines, and of any additional text shown at the top of this section, is entirely up to the user who either creates or modifies a tool using the tool builder.
The clinical utility of the one condition tool could be summarized as follows: First, the one condition tools allow the complete replacement of ANALYTE cutoffs with a single CONDITION-specific threshold of clinical significance, which is the 1st percentile of all scores; second, isolated, random abnormalities that do not fit in the pattern defined by the plot by condition are filtered out as analytical noise. Third, the interpretation of a profile is driven by the %ile rank of a patient in comparison to all true positive cases in the R4S database, a type of clinical validation that is dynamic, not static, as it evolves practically on a daily basis. Finally, a tool has been released into production for every condition detected by tandem mass spectrometry with a count of submitted cases equal to or greater than 3 (ranging between 3 and 1,301 cases).
It is time now to move on to the second type of tool, the dual scatter plot. Differently from the one condition tool, one that seeks a yes or no answer, the dual scatter plot has the capability to resolve the differential diagnosis between 2 conditions with similar and overlapping biochemical phenotypes.
Once again, we use VLCAD and VLCAD carrier status as the model to demonstrate how the dual scatter plot actually works. A side by side comparison of the 2 plots by condition is helpful to recognize the crucial role of 2 analytes, C14:1 and C12:1, and one ratio, C14:1 to C12:1.
This slide shows a simplified version of the plot by condition, where only the 3 markers highlighted in the previous slide are displayed. Using a plus/minus evaluation, the plots show that the disease range of C14:1 is markedly elevated in VLCAD (3 pluses, no overlap with the reference range) but also moderately to markedly elevated in VLCAD carriers (shown as 2 pluses). Surprisingly, C12:1 behaves differently, with a higher range and consequently less overlap in carriers. Putting these findings together, it becomes apparent that the C14:1 to C12:1 ratio is the key differentiator between the 2 conditions.
To appreciate how the dual scatter plot works to segregate VLCAD and VLCAD carriers, shown here in bright red and orange, respectively, it is useful to first recall the underlying rules of the one condition tool. In the case of a high marker, results below the upper limit of the reference range do not contribute to the score. On the other hand, any result above the 99th percentile of the respective reference range generates a score that is proportional to the degree of penetration within the disease range that does not overlap with the reference range.
In the dual scatter plot, the rules are quite different. First, the relationship to the reference range becomes irrelevant as the comparison now takes place between 2 disease ranges. If the result falls within the range of overlap, there is no score modification. However, if the result is either below or above the area of overlap it triggers a score modification that is proportional to the degree of separation from the disease range of the other condition.
The dual scatter plot is actually the combination of 2 tools, one that targets any non-overlapping result to increase the score for VLCAD and to decrease the score for VLCAD carrier. The other tool, shown on the right side, operates exactly in the opposite way. A result within the overlap range triggers no modification. A result above the overlap range increases the score of the tool on the left and has the opposite effect on the tool on the right. A result below the overlap range reverses the impact on the 2 sides. So, a completely normal C14:1 to C12:1 ratio is actually very informative to achieve the desired differential diagnosis, even if the same result would not trigger any score in a one condition tool for either condition.
Another characteristic of the dual scatter plot is a different way to express the calculated scores. Instead of absolute values, the Y-axis uses a minimum-maximum normalization so that all score are kept within 0 and 100. Each result is calculated by subtracting from the score the lowest of all scores, dividing it by the range of values (highest minus lowest), and multiplying it by 100. As shown on the right side, this formula preserves the relative distance between values and is ideal to achieve consistency among tools comparing any 2 conditions with different numbers of informative markers.
The impact of these rules is best demonstrated by showing what happens to the 2 sets of scores when the rules are applied. First, we will consider the tool that is designed to favors VLCAD and penalize VLCAD carriers, all shown here as clusters of yellow circles before the rules are applied. In a Min-Max mode the Y axis now goes from 0 to 100. The scores and more importantly the segregation between the 2 groups changes quite dramatically after the rules have been activated.
The effect in the other tool, the one favoring VLCAD carrier status is show in this slide. Again, there is a significant improvement in separation between the 2 groups.
When the 2 tools with active rules are combined, the final result is a dual scatter plot.
The output of the dual scatter plot is a visual separation of the combined scores in four quadrants. The lower right quadrant includes the cluster of cases with condition 1, those with high score in one tool and a low score with the other. The upper left quadrant includes the scores of cases with condition 2. A score located in the upper right quadrant is equivalent to an inconclusive result, meaning that both conditions are still possible. Finally, a score in the left bottom quadrant excludes both conditions. When used to investigate an unknown case, the coordinates of the combined scores of that particular case are shown as a red diamond.
Access to the dual scatter plot is from the same page that lists all post-analytical tools.
Tools are labeled in a manner consistent with the nomenclature of the one condition tools: name of condition 1, name of condition 2, version number of the tool for condition 1, version number of the tool for condition 2, a D or a U within squared brackets to indicate the inclusion of at least one marker that is affected by the use of a derivatized or underivatized method, and the date when the combined tool was created.
Data entry is identical to the one condition tool, either an automated or manual process complete by clicking the calculate icon.
The dual scatter plot available on the R4S website to distinguish between VLCAD cases and VLCAD carriers is shown here. The score of this case, shown as a red diamond, is clearly located within the VLCAD cluster, far away from either the VLCAD carriers or the inconclusive and negative quadrants.
In addition to the plot, the report shows the 2 calculated scores, the respective percentile rank, and the count of cases in each group.
The report of the dual scatter plot also includes interpretation guidelines for each of the quadrants. In our experience, the Dual Scatter Plot has greatly improved the differential diagnosis between VLCAD and VLCAD carriers, preventing many unnecessary referrals and follow up testing. Inconclusive results should be treated as indicative of VLCAD and receive appropriate follow-up testing.
This is the conclusion of the third part of the R4S series of Mayo Medical Laboratories Hot Topics. In part IV, we will introduce the 2 high throughput data entry portals available in R4S to perform the simultaneous calculation of all score for a single case, the all conditions tool, or for large batches of cases, for example a 96 well plate. This second functionality is named the tool runner.
Please do not hesitate to contact us if you have any questions or requests related to the content of this presentation. Thank you very much for your attention.
Piero Rinaldo, M.D., Ph.D.
Professor of Laboratory Medicine and Pathology,
Mayo Clinic, Rochester, Minnesota
Thank you for the introduction. This presentation is the first portion (part A) of the fourth segment of the series describing the products and clinical tools of a newborn screening quality improvement project called Region 4 Stork, or R4S. The title of this presentation is “High-throughput data entry portals”. During editing this topic has been divided in two portions to remain within the required time limits of this type of presentation.
I have a disclosure to make: a provisional patent application related to some of the content of this presentation has been submitted by Mayo Clinic. The title of the presentation is “Computer-Based Dynamic Data Analysis.”
This presentation continues the overview of the second generation of R4S tools, this time focusing on how to use them effectively in a daily laboratory practice where convenient uploading and rapid, large scale data analysis are highly desirable.
In the previous segment, part III of this series, the tool builder and the two most commonly used tools, the one condition tool and the dual scatter plot, were introduced.
Although it was described how these tools are produced using the tool builder, there was no information about their use in a laboratory setting. It was done this way to underscore their clinical utility to a user rather than a “producer” of laboratory results. Indeed, the one condition tool and the dual scatter plot can be described as the CLINICIAN TOOLS, tools that can be clinically useful literally at the bed side of the patient after laboratory results have been interpreted and reported. In this segment, the focus is instead on the LABORATORY TOOLS, the tool runner and the all conditions tool.
We begin with the tool runner. This tool allows the simultaneous evaluation of all conditions with an active tool for multiple patients in any number from a few to thousands, typically one 96-well plate at a time.
What is the tool runner? The tool runner is a portal to upload to the R4S website whole batches of raw data after conversion to a .csv (comma separated value) file. This type of file can be generated routinely, and automatically, by virtually any operating system of commercially available tandem mass spectrometers, and also most types of laboratory instrumentation. The tool calculates automatically every possible score for each case in a batch using all released one condition tools, or a smaller, user-selected panel. The tool runner generates a report of all instances with a score greater than the 1st percentile of the scores of all cases with a given condition. In other words, a score within the range of values obtained for the population of true positive cases.
This slide summarizes the process from raw data to an actionable report: the instrument software generates a csv file, the file is uploaded to the tool runner on the R4S website, all one condition tools are run simultaneously, the summary report of informative scores is produced. To emphasize the user-friendly nature of this process, we call it click click done.
The apparent simplicity of this process from the viewpoint of an average user is quite a contrast to the underlying complexity and magnitude of the data being analyzed. A typical 96-well plate includes approximately 90 patient samples, depending on the number of controls added to each plate (in this case 5, the first 3 and the last 2 wells shown as blue circles). The total number of analyte results and calculated ratios for a plate is also variable, but in most cases exceeds 100. All things considered, a single plate analyzed for amino acids and acylcarnitines by tandem mass spectrometry routinely generates more than 10,000 results. The magnification of a small portion of this spreadsheet allows a better understanding of the raw data structure.
In this instrument-generated spreadsheet, one patient per row, the following elements are included: the overall file name of the batch (please note the file type .WIFF is the file format of one type of commercial instruments in use in our laboratory; other common types are, for example, “.RAW” and “.D”); the sequence number (here called the sample index); the specimen ID number that is assigned by the main laboratory information system; the last name of the patient, not shown here for obvious reasons; the sample type (“unknown” indicates a routine first specimen); and finally the analyte abbreviated names and actual results.
2 of these elements, the specimen ID and the patient name, are considered patient health information, or PHI, and as such are not suitable to uploading to a web-based application even if password protected like R4S.
For this reason, instrument softwares can be programmed to generate a different type of file using the .csv file format. As a reminder, a comma-separated value file stores data in a plain-text form. Plain text means that the file is a sequence of characters separated by a literal comma or a tab. As ratios are calculated by the R4S tools, only analyte values need to be included, reducing the cumulative number of results to less than half of the original file. Again, magnification of a small portion of this spreadsheet allows a better understanding of the raw data structure.
The data elements of a .csv file are the following: the analyte abbreviated names; the sequence number; the analyte results and a very important new element, the LOINC codes. LOINC stands for Logical Observation Identifiers Names and Codes which is a universal code system for identifying laboratory and clinical observations. As stated on the homepage of the website maintained by the Regenstrief Institute at the University of Indiana, LOINC has standardized terms for all kinds of observations and measurements that enable exchange and aggregation of electronic health data from many independent systems. A LOINC code is unique to a combination of component, system (sample type), scale and unit of measurement. More information on the application of LOINC codes in newborn screening can be found in this publication.
Going back to the csv file, the final output is void of any PHI, individual cases are identified only by their sequence number in the batch, an information that cannot be traced back to a specific patient.
When ready, the .csv file needs to be uploaded to the data entry portal of the tool runner. A user needs to login on the R4S website, access the post-analytical tools page, and click the tool runner link.
The selection page of the tool runner consists of three sections: they are, from top to bottom, select a tool, tools selected to run, and select data file.
Not all functionalities are accessible to all users. For example, the option to select a not yet released tool for testing and validation purposes is limited to those users who have been granted access to the tool builder. The remaining options in this section are relevant to a situation where a user is interested to run a subset of tools instead of the default action, which is to run all of them together. If, for example, a MN user wants to run a single tool created to identify MCAD carriers, he or she would first select fatty acid oxidation as condition type, then select the desired condition, tool type, and tool version from the drop down menu. By clicking the “Add” icon the chosen tool is activated in the next section.
In the next section, Tool selected to run, the opposite action can be taken, which is to remove one or more of the previously added tools. To continue using the customized subset, the user must select “Run selected tools” instead of the default “Run all single tools”. Two additional functionalities are available, one is to include in the report all the scores greater than zero but still below the selected threshold of clinical significance, the other is to convert the standard report into an unprotected Excel file ready to be saved on the user’s computer.
In the third and final section, Select data File, users have the option to select either the file format compatible with derivatized profiles, shown as D within squared brackets, or the alternative format that works with underivatized profiles, shown as U within squared brackets. The correct format should be selected and displayed automatically if a user with read & write access from the same site had answered the series of questions in the data submission/participant profile menu. If a choice was made in this window the format is adjusted accordingly in the tool runner. If an answer had not been provided yet, the default format is D. A user of an underivatized method from a site with an incomplete profile would not be prevented from using the tool runner, but an error message would appear explaining that the incorrect LOINC codes were submitted for some analytes.
When the display shows the “no file chosen” message, the uploading of a batch file is quick and easy once users have created on their computer desktop a shortcut to the server location where the instrument-generated files are stored. A click on the link brings on subfolders by year, by month, and by day. Each subfolder corresponds to a 96-well plate and it includes the .csv file ready to be uploaded to the tool runner.
Another seldom needed feature is the option to eliminate from the calculation of scores additional wells of the 96-well plate in addition to the first three and the last two which are used for quality control purposes. Again, many of these features are included to provide the greatest possible flexibility to meet diverse needs of a diverse number of users, but in reality the processing of a batch should not take more than a few seconds to access the tool runner, select a file, and click “Run Tools”.
After clicking the run tool icon, a report is generated in just a few seconds. The header shows how many profiles were processed (91) and how many scores were calculated (3731). The .csv file name and the batch ID of the original instrument-generated file are also included to facilitate tracking. Before discussing the report in more details, it is important to explain how the system is designed to react to incomplete data set in order to prevent the incorrect assumption that a condition tool was not informative because of one or more results were actually missing. In this instance, everything was in place, so no alert message was displayed to the user. On the other hand, the uploading of the same file after a single value had been removed randomly, in this example just the numerical result of the acylcarnitine C5:1 in row 23, triggers a report with a clear indication of the underlying problem. The header now includes a new descriptive element, the “Not run count”, and a table listing the specific tools that were inactive. If the C5:1 value is removed from all rows.
The error message is phrased accordingly, with the addition of a note reminding the user that any inactive tool could be modified to be site-specific and not inclusive of the missing analyte. As a reminder, tool customization will be the main topic of the next presentation.
Going back to the report, when the system is set up properly truly all that it takes to generate a report is three clicks of the mouse. The report shows that out of the initial 91 cases there were only three with informative scores. The first one, sequence ID 26, triggered a score for the VLCAD carrier tool, but not for true VLCAD. For this reason, this finding could be discarded without further evaluation. The VLCAD carrier tool is site specific, only MN users can see it. Indeed, it is not available to all users because it generates a score frequently and could have the opposite effect of the real goal of the tool runner, which is to limit the number of unnecessary referrals. The second case, sequence ID 61, indicates multiple amino acid elevations that fit the pattern of total parenteral nutrition. In our experience, this result is sufficient to classify it for what it is, a nutritional artifact of no clinical significance that requires no referral to follow up and not even the collection of a repeat sample. In this batch, there is a SINGLE case that requires closer evaluation, the patient with sequence ID number 84.
The tool runner report shows an informative score for three conditions, VLCAD deficiency, 3-hydroxy acyl-CoA dehydrogenase deficiency combined with Trifunctional deficiency (shown as LCHAD / TFP; from this point forward, these conditions will be described simply as LCHAD), and again VLCAD carrier status. Selection of the blue link for VLCAD brings up the data entry window with all the result fields already populated. Clicking the Calculate icon at the bottom of the window produces the one condition tool for VLCAD deficiency.
This slide shows the connection between the content of the tool runner report and the elements included in the score report of the one condition tool. The three elements that cross over are the actual score, calculated against the entire population of true positive cases and the percentile rank among those cases. The third element comes from the interpretation guidelines, and that is the interpretation group (score 30 to 50) that includes the calculated score (45).
It is worthwhile to show a quick reminder of the methodology used to define the interpretation groups in a consistent and objective manner. In the tool builder, the verification of a new tool before release into production is based on two steps called IMPACT and RUN. In the first one, the testing of different rules generates a report called Condition Score Ranges. The green background signifies that the rules did have a positive impact on the scoring. The interpretation guidelines of all R4S tools are based on three percentile values, the 1st, the 10th, and the 25th percentiles of the condition score range. For consistency, all threshold values are rounded to the nearest multiple of 5. A score below the 1st percentile, in this case a score less than 30, is considered not informative and therefore will not be included in the tool runner report unless the user activated the option to see not informative scores. A score between the 1st and the 10th percentile, or between 28 and 51, rounded to 30 and 50, respectively, is considered informative with a textual representation of being POSSIBLY indicative of VLCAD deficiency. As the score goes higher, between the 10 and the 25th percentile, it is represented as LIKELY indicative of VLCAD deficiency. Finally, a score greater than the 25th percentile is VERY LIKELY to be consistent with a biochemical diagnosis of VLCAD deficiency. No additional categories are deemed necessary as scores above the 25th percentile should be increasingly self-evident even when using the most conservative cutoff values. The tool builder allows the inclusion of a header that could be used for disclaimers. In the newborn screening tools, this option is used to warn users that R4S tools has been validated only for neonatal blood spots, collected before 10 days of age.
Going back to case number 84, this slide shows the three tools with informative scores. This outcome if far from being indicative that further action is required. Instead it should be framed like a differential diagnosis between VLCAD and VLCAD carrier and a verification that the score for LCHAD deficiency, barely above the 1st percentile, is indeed sufficiently significant to warrant a referral to follow up.
As discussed extensively in previous segments of this series, the degree of overlap between reference and disease ranges is the conceptual foundation of the post-analytical tools. In LCHAD deficiency, most of the informative markers, namely C16 and C18 hydroxy long chain species and related ratios, are present under normal circumstances at very low levels. Many participating sites have listed a zero value for their 1st and 10th cumulative reference percentiles, a situation that translates in the peculiar behavior represented in the plot by condition, especially at the low end. Using these skewed reference ranges, this case yields a score barely above the threshold.
Not all sites are affected by this problem, and indeed Minnesota is not one of them. In a case with a borderline score for LCHAD deficiency, MN users are trained to switch to our own percentiles, a change achieved by a single click in the data entry window.
The impact of this customization is shown here. The overlap between the reference and disease ranges of these analytes is reduced significantly. The calculated score (6) is now well below the 1st percentile of the LCHAD score range, and therefore can be interpreted with confidence to be a not informative (in other words, normal) result.
This simple process allowed the exclusion of LCHAD deficiency, leaving unresolved only the differential diagnosis between VLCAD deficiency and VLCAD carrier status. As shown in the previous segment, this can be promptly achieved by switching to the dual scatter plot.
The plot places convincingly this case within the VLCAD carrier cluster. In summary, the post-analytical interpretation of the results of a 96 well plate, 91 patients, can be finalized in less than one minute of work by a user trained to navigate the R4S website and the tools wherein. This process is rapid, paperless, and most importantly it provides a mechanism to maintain consistency among different users who may have vastly different expertise, from just a few months to years if not decades of experience.
To recapitulate the clinical utility of this functionality, the Tool Runner allows the simultaneous analysis of large batches of NBS data. While testing of one plate at a time is a logical approach in a routine situation, a .csv file where the data of 20 96-well plates were merged for testing purposes has been analyzed successfully. A database on approximately 60,000 cases has been processed after the selection of a single one condition tool. As a reminder, the tool runner calculates a score with every available one condition tool and generates a summary report of all informative scores. On the other hand, it can be applied to either a single tool or to any desired combination of general and/or site specific tools. In our experience, it is not uncommon to see reports like this one, where an entire plate could be resulted as negative when not a single one of the calculated 4095 scores was informative. With the understanding that some users might not be completely comfortable with this type of outcome, it is possible to expand the report to include all scores that were anywhere above zero but still below the threshold of clinical significance. Finally, the report of the tool runner can also be exported to Excel for documentation, research, quality assurance, and system evaluation purposes. Since the launch into production of the tool runner, a frequently asked question has been “Is anybody (beside MN) using it”? That is a legitimate question that we are glad to answer.
The tool runner became available to R4S users in November 2011 after being presented to the final R4S user meeting in San Diego. This activity was made possible by the HRSA Regional Collaborative grant that ended its second cycle in May 2012. It was utilized 79 times in December 2011 by MN and two other programs. In 2012, however, the tool runner was deployed more than 6000 times, 700,000 profiles were tested and almost 17 million scores were calculated. In the first two months of 2013 more than 5 million scores have been calculated, an 80% increase when expressed as average utilization per month. To date, the tool runner is used regularly by 21 programs; the majority of them are located outside of the United States.
This is the conclusion of the first portion of the fourth presentation of the R4S series. In the second portion, part B, the second type of high throughput data portal, the all conditions tool, will be presented.
Please do not hesitate to contact us if you have any questions or requests related to the content of this presentation. Thank you very much for your attention.
Piero Rinaldo, M.D., Ph.D.
Professor of Laboratory Medicine and Pathology,
Mayo Clinic, Rochester, Minnesota
This presentation is the second portion (part B) of the fourth segment of the series describing the products and clinical tools of a newborn screening quality improvement project called Region 4 Stork, or R4S. The title of both presentations is “High-throughput data entry portals”. During editing this topic has been divided in two portions to remain within the required time limits of this type of educational presentation.
The final tool to be presented here is the all conditions tool.
Like the tool runner, this tool also covers all conditions with an active tool, but it does it for just 1 patient at a time.
The second major difference is the type of report. The all conditions tool generates a report in the form of a composite graphic, not a table. It does calculate every possible score, and it reports on any missing data required to calculate a score with a particular tool. Finally, in this format all tools can be readily accessed, even if the score was zero.
A single case file can be extracted from the batch file formatted for the tool runner. From that point on, the process is exactly the same: uploading to the data entry portal, calculation of scores for all one condition tools, and creation of a report. This is, of course, not the only way to access the all conditions tool.
Just like all other post-analytical tools, once a user has logged in the R4S website the all condition tool can be accessed from the link on the home page and then selecting the link to the all conditions tool.
Moreover, direct access is provided from the report of the tool runner for every case with at least 1 informative score. This tool can also be used retrospectively, for review of true positive cases already posted on R4S. From the main menu bar below the website blue banner a user can select the Tool and Reports menu, site tools (in this case Minnesota), and the True Positives menu. A selection window (only the top section of this window is being shown here) allows to select a condition and then a specific case. At the top of the next window a link to the all conditions tool is displayed. This option is also provided in the True Positives data submission menu once a new case has been added to the R4S database.
This slide summarizes the elements and optional choices available on the selection window of the all conditions tool.
The first dialog allows the selection of condition types. The default setting is to show all types, but a user interested to see, for example, only fatty acid oxidation disorders can easily unclick the 2 other groups and generate a report showing only the scores of tools for this type of conditions.
The distinction between primary and secondary targets is based on the recommended uniform screening panel, or RUSP. It was established in 2006 by an expert panel assembled by the American College of Medical Genetics with funding from HRSA, the same agency that funded the R4S project. In the following years, this panel was first reviewed and approved by the Health and Human Services Secretary Advisory Committee on Heritable Disorders in Newborns and Children. The committee then made a recommendation to the HHS Secretary, Dr. Kathleen Sebelius. In May 2010 the Secretary agreed to make RUSP the national standards for newborn screening programs in the United States. Like in the case of condition types, a user can choose to display only the conditions included in 1 group, for example only primary targets.
The selection of score type drives the visual appearance of the lower section of the report. The default is the MinMax type, the 1 that was discussed extensively in the analysis of the dual scatter plot.
Briefly, the display of the same data using the three types is shown here: MinMax, Regular, and Z-score. The Z-score is based on the calculation patient value minus the average of the population divided by the standard deviation of the population. The z-score is multiplied by 100 and a value of 500 is added. This transformation expands the range of scores so that 95% of them fall between 300 and 700. The addition of 500 shifts any score reduced by differentiator rules to a positive number. The application of z-scores will become increasingly more important once the next version of the software, for convenience referred to as “version 2.0”, is completed, and will be further discussed in part VI, the last segment of this series.
If the all conditions tool is accessed without previous processing of a batch file, it might be necessary to upload a single case .csv file. A macro script to create such file is available to all users and could be customized easily to match the format of raw data files generated by virtually any type of instrument.
Once a file had been uploaded, or transferred from another tool, it is always possible to review the data by clicking the banner “Show analytes”. This is also the process to manually upload data, which is an available option but could be very time consuming and therefore it is not recommended. Once the review and/or data entry is completed, the window is compressed by clicking “Hide Analytes”.
After any desired modification of the settings and inclusion of a data set, “Run Tools” generates a visual report like the 1 shown here.
The elements included in the All Conditions Tool Report are the following: file name, percentile rank of calculated case scores, the range of condition scores according to the selected type, a color legend for the diamond symbols representing the scores, and, if applicable, a list of missing analytes, if any (none is listed in this case).
This slide shows a few examples of the information provided by the all conditions tool: it could be a straight answer, but for a rare condition like Malonic acidemia that most programs have little or no previous experience with, and therefore could be at risk of being missed by an improperly set cutoff value. Alternatively, the report could reveal a situation of moderate complexity, like this case where there are informative scores for maternal vitamin B12 deficiency and different types of inherited Methylmalonic acidemias. The complexity is considered moderate because all these conditions require the same 2nd tier test and confirmatory testing in plasma and urine. The tool may provide a completely unexpected answer, like this case who is affected with a remethylation disorder, a group of at least 3 conditions not yet included in the recommended panel but likely to be added sometime soon. One of the 2 main features of the biochemical phenotype of these conditions is a low concentration of the amino acid methionine, and related ratios. It is likely that many cases like this will continue to be overlooked because many laboratories do not even have a low cutoff for this amino acid, so a result near zero would still be considered to be “normal”. In the R4S database, 124 laboratories have posted a high cutoff for methionine, but only 66 of them also have indicated their use of a low cutoff value. This is unfortunate because the availability of a highly dependable 2nd tier test for the other characteristic finding of this disorder, hyperhomocysteinemia, can drive the specificity of screening for these disorders to virtual perfection, with no false positive outcomes. The next example is the same case that was analyzed in a previous segment of this series to demonstrate the clinical utility of the dual scatter plot, one of the so-called first generation tools: the comparative evidence provided by the all conditions tool is compelling and should prevent the referral of a case with moderate elevations of branched chain amino acids when the underlying explanation is total parental nutrition (TPN) instead of MSUD. Finally, the most important example is the 1 shown here: a completely negative profile that exemplifies the extremely common situation where random elevations of one or more analytes, something we can describe as random analytical noise, could trigger an unnecessary referral even when the overall profile is not consistent with any known condition.
The clinical utility of the All Conditions tool can be best described as the ability to serve as an effective gateway to a comprehensive and unbiased differential diagnosis of all possible conditions. In the report, all scores (shown as diamonds) and ranges are active links to the respective one condition tool. This design allows a user to explore in rapid sequence multiple one condition tools, even those with uninformative scores, which are shown as gray diamonds. The All Conditions Tool has been utilized 3,389 times in 2012, by more than 100 users worldwide. Similar to the increased awareness experienced with the tool runner, in 2013 the average utilization per month has increased by 67%.
This is the conclusion of the second portion of part IV of the R4S series of Mayo Medical Laboratories Hot Topics. In part V we will describe how a user with access to the tool builder can accomplish a site-specific customization of the post-analytical tools, particularly of the one condition tools and of the dual scatter plot, and their seamless incorporation in the high throughput tools we just presented in this segment.
Please do not hesitate to contact us if you have any questions or requests related to the content of this presentation. Thank you very much for your attention.
Piero Rinaldo, M.D., Ph.D.
Professor of Laboratory Medicine and Pathology,
Mayo Clinic, Rochester, Minnesota
Thank you for the introduction. This presentation is the fifth segment of a 6-part series describing the products and clinical tools of a newborn screening quality improvement project called Region 4 Stork, or R4S. The title of this presentation is “Site-specific customization of post-analytical tools”.
I have a disclosure to make: a provisional patent application related to the content of this presentation has been submitted by Mayo Clinic. The title of the application is “Computer-Based Dynamic Data Analysis.”
After completing the overview of the different types of interpretive tools and data entry portals, the topic of this presentation is site-specific customization of the post-analytical tools.
There are 2 types of tool customization in R4S: the first 1 is the option to switch from cumulative to own percentile values in the data entry window of the one condition tools. The clinical indications of this action have already been discussed in a previous segment of this series. The main topic of this presentation is the permanent modification in the tool builder functionality of a released tool that is available to all users. This option is driven by the need to fit the diverse analyte panels implemented by individual sites.
The condition models selected for this presentation are 2 inborn errors of metabolism of the proximal urea cycle: ornithine transcarbamylase deficiency, abbreviated as OTC, and carbamylphosphate synthetase deficiency, or CPS. The first of these conditions is a relatively common X-linked disorder; the other is a rare autosomal recessive disorder. The defining clinical manifestation of OTC and CPS is found in the occurrence of acute, potentially life-threatening episodes of hyperammonemia. The biochemical phenotypes of OTC and CPS are not identical but share the finding of low plasma concentration of the amino acid citrulline. Although this abnormality is readily detectable in the amino acid profile and a variety of treatment options are available, OTC and CPS were not included in the recommended uniform screening panel because at the time of the proceedings of the expert panel, between 2004 and 2006, it was felt that there was no screening test with adequate sensitivity and specificity.
However, elevated citrulline is regarded as a reliable marker for several conditions shown here in the Plot by Target Range tool. They are, from left to right, citrullinemia type I, pyruvate carboxylase deficiency, citrullinemia type II, also known as citrin deficiency, argininosuccinic acidemia and maternal citrullinemia type I. 2 of them, citrullinemia type I and argininosuccinic acidemia, were included in the recommended uniform screening panel, citrullinemia type II is one of the secondary targets. For these reasons, the determination of citrulline is included in the vast majority of analyte panels adopted by screening laboratories. The differences between the reference range, the cutoff target range and the respective disease ranges can be better appreciated by an expansion of the Y-axis limited to the boxed area highlighted here.
To date, 121 of the laboratories participating in the R4S collaborative project have selected a high cutoff value for citrulline. Overall, there are more than 500 cases in the true positive database, the vast majority of them is predictably affected with the 2 primary targets, citrullinemia type I, almost 300 cases, and argininosuccinic acidemia, almost 150 cases.
As mentioned in a previous segment of this series, the plot by marker is a tool that allows an unbiased and comprehensive view of disease ranges for a single analyte, in this case citrulline, in all conditions regardless of their clinical significance. On the right side of the plot, it is evident that there are conditions with disease ranges significantly below the reference range.
Going back to the plot by target range, but this time below the reference range, it is possible to recognize the conditions with low citrulline: OTC and CPS, of course, symptomatic female carriers with OTC deficiency, and another rare condition, ornithine aminotransferase deficiency. The total number of OTC and CPS cases combined exceeds 100. On the other hand, only 64 laboratories have selected a low cutoff value. In other words, approximately half of the labs actively monitoring citrulline at the high end apparently would not act on a low value even if the concentration measured in a patient was near zero.
To address this less than ideal situation, the American College of Medical Genetics released in 2012 a new act sheet addressing the finding of decreased citrulline as an actionable marker for the detection of proximal urea cycle disorders in neonatal dried blood spots.
This slide shows the Plot by Condition for OTC/CPS. The display is limited to informative amino acids, and therefore only citrulline is displayed. Although there is clear separation between the reference range, shown as a green shade, and the disease range, shown as the red box, the act sheet correctly underscores a warning that a decreased citrulline concentration alone, in most instances is NOT informative per se and could lead to an excessive number of false positive cases.
The diagnostic utility of low citrulline could be much improved by giving proper consideration to at least seven different ratios. 2 of them, methionine to citrulline and citrulline to phenylalanine, are commonly used as markers of conditions with abnormal levels of methionine and phenylalanine. Citrulline is often used to calculate ratios because this amino acid is the least influenced by total parenteral nutrition and other types of interference. 5 additional ratios are routinely calculated in R4S for the specific purpose to recognize cases with proximal urea cycle defects: these are, from left to right, the alanine to citrulline ratio, the glutamate to citrulline ratio, the glutamine to citrulline ratio, the ornithine to citrulline ratio, and the citrulline to arginine ratio. One of them, the glutamate to citrulline ratio, is the one with the lowest degree of overlap between reference and disease range, only 6.7%. Naturally, the availability of 8 informative markers allows the creation of a post-analytical interpretive tool for OTC and CPS, indeed this tool was one of the Excel-based prototype tools introduced on the R4S website since 2009. The current version is number 16 and was last updated in January 2013.
Just like all other tools, the amino acid concentration values can be uploaded or entered manually in the data entry window and the tool is generated after clicking the Calculate icon. The score comparison plot is highlighted here to call attention to the fact that this particular case is actually the one with the lowest score among the 11 cases, a mixture of retrospective and prospective findings, contributed to R4S by the Minnesota program.
The score section and the score interpretation guidelines of this case are highlighted in this slide. The calculated score, 76, is at the upper end of the “possibly OTC” range (a score between 25 and 80). The all condition tools is also clearly suggestive of the correct diagnosis (diamond placed at the 9th percentile rank among all 44 cases) even if the percentile rank in comparison to the 11 Minnesota cases is listed as zero. So, the tools appear to be working properly and therefore it is legitimate to ask the question: where is the problem?
The problem is clearly outlined in the last sentence of the introductory paragraph of the ACMG act sheet. It reads “Note that some newborn screening laboratories do not report decreased citrulline OR ABNORMAL AMINO ACID RATIOS”. The final comment refers to a different situation because the primary analytes required to calculate a ratio to citrulline are not being measured at all.
For example, the same profile of the true positive case described earlier would not appear in the all conditions tool for a laboratory we will call “Lab 1”. The tool, however, would include a warning that the OTC/CPS tool did not generate a score because one required analyte, ornithine, was not measured by this particular laboratory.
The same outcome takes place for another site, called Lab 2, but the failure to calculate a score is compounded by a second missing analyte, glutamine in addition to ornithine.
Laboratory 3 is missing 3 of the required amino acids, ornithine, glutamine and glutamate, notably the latter is the one required to calculate the most informative marker.
Laboratory 4 does measure ornithine but glutamine, glutamate and alanine are not included.
Finally, Laboratory 5 does not measure ornithine, glutamine, and arginine.
This table summarizes the profiles described in the previous slides: these labs have different amino acid panels, but all of them for one or more omissions are prevented from using the OTC/CPS tool available on the R4S website.
It should be noted that these are not hypothetical situations, they all are true examples selected from the reference percentiles and cutoff profiles of active R4S participating sites. To place this evidence in a more general context, this slide shows the percent of laboratories measuring individual amino acids in addition to phenylalanine, which is used here to normalize percentages. In other words, the number of labs with either calculated percentiles or an active cutoff for phenylalanine is equal to 100%. 7% of these laboratories do not measure methionine. This finding likely reflects the proportion of sites that currently use tandem mass spectrometry to screen exclusively for PKU and MCAD deficiency. A slightly higher proportion, 9%, does not measure citrulline; the difference is likely to reflect the number of labs who have added testing for homocystinuria to a limited panel that already included PKU and MCAD. When arginine is considered, 25% of the laboratories do not measure it. Alanine is close to be evenly split, approximately 50/50, but ornithine, glutamate and glutamine are indeed NOT measured by an overwhelming majority of labs. In summary, 40 to 90% of the participating laboratories do not measure routinely these clinically useful amino acids.
The situation becomes even more concerning when considering the proportion of labs with no evidence, percentiles or cutoff values, of using the 7 ratios that are needed to calculate a score with the OTC/CPS tool, ranging from 48 to 92 percent. Notably, the glutamate to citrulline ratio, the most informative marker for OTC/CPS in neonatal dried blood spots, is NOT measured by 88% of the laboratories participating to the R4S collaborative project. Obviously, there is a lot of room for improvement.
A defining function of the R4S collaborative project is to collect and visualize objective evidence to improve awareness among users of the importance to utilize the full complement of informative markers for every condition. Said that, and to compensate for the pervasive lack of consistency, the R4S tool builder is designed to allow the creation and release of tools with modified marker panels that are site-specific and accessible only to users affiliated with a single site. This process is remarkably simple and is summarized in the 5 steps shown in this slide: create a copy of a general tool, edit markers, verification of score percentiles, editing of interpretation guidelines, and release into production. A trained user can complete this process literally in just a few minutes, using a protocol that is described in detail in the following slides.
In the tool builder menu, the selection of the tool to be copied takes place in the list/release window. The list is quickly limited to OTC/CPS by selecting the condition type and the desired condition. There are 3 options available for an active tool: view, archive, and copy. The other icons, edit, release, and delete, are inactive and shown in a lighter color font because they are not applicable to an active tool. Next action is to select COPY.
This action brings up a new dialog window titled “Create Copy for Participant”. The default setting is “Available to All”, to change it just select the drop-down menu to display a list of all active participating sites.
For the sake of this presentation, the list has been modified to show only the 5 sites previously identified as Lab 1, 2, 3, 4, and 5.
If lab 1 is selected, the copy of the tool will be specific to this site and be accessible only to users affiliated with Lab 1.
This selection is finalized by clicking “Save”.
The list now shows the addition of a new tool with a clear indication it is unique to Lab 1. The same action is repeated 4 more times for the other labs, and the list expands accordingly. Notably, all these tools are still invisible to the users of these sites and will remain as such while the necessary modifications take place before being released into routine production.
When a tool is not yet released, the options that are active in the tool builder are “Edit”, “Release”, “Copy”, and “Delete”. The next step is to select Edit.
In the Edit Marker page, high and low markers are listed in separate tables. The columns of these tables are, from left to right, the name of the marker, the count of results available in the database, and the degree of overlap between reference and disease range. 4 other columns summarize additional attribute of the markers and will not be discussed in detail here. If desired, a complete explanation is accessible in the Documentation menu of the R4S website, section “Presentations”. The last column, with the header “Select” is the one where the inclusion of a marker in the tool is determined.
These slides focus on the Select column and shows only the analytes included in the general tool, the one available to all sites. The modification required to make a site-specific tool for Lab 1 is accomplished simply by unclicking the corresponding checkbox, followed by selecting “Save Changes”.
The same process is followed to customize the analyte panels for the other 4 laboratories, again completed by selecting “Save Changes”.
The next step is the verification of the percentile score ranges, and the consequent adjustment of the score interpretation guidelines. As a reminder, the scores corresponding to the 1st, 10th, and 25th percentiles, rounded to the nearest multiple of 5, define the boundaries between the interpretation guidelines: a score lower than the 1st percentile is considered not informative, a score between the 1st and the 10th percentile is informative, but is labeled as “possibly”. Any score greater than the 10th percentile, in this case a value of 80, is increasingly more and more likely to reflect a true positive case. As stated before in this series, scores that exceed the bottom quartile, in other words greater than the 25th percentile of all scores calculated for true positive cases, become increasingly self-evident and should never constitute a diagnostic challenge.
Predictably, the deletion of one or more markers has a direct effect on the scores and the corresponding percentiles, and therefore need to be adjusted in each site-specific tool. In the case of lab 1, the count of cases with all required markers actually increase from 44 to 47 (the total number of cases with at least one value is actually 104), the changes of percentile values are almost negligible. That is not the case for the other tools. Even if the count increases as high as 75 (a 70% increase), the 1st percentile for Lab 2 dropped by more than 50%, from 27 to 12, and it goes even lower, 6, for the other 3 laboratories, lab 3, 4 and 5. Basically, this is an indication of the strength of the site-specific tools in comparison to the general one available to all sites.
The next and final step is to select the icon “Release” for each site-specific tool.
After that action is completed, all tools become available to any user of those sites and are automatically deployed each time the tool runner and/or the all conditions tool are utilized.
To recap the process to create a site-specific tool, 5 simple steps are all that is needed: copy tool, edit markers, calculate scores, update guidelines, and release of the tool.
The outcome of this process is shown here. Before customization, lab 1 could not generate a score for this case. After the customized tool was released, the outcome was clearly informative, with a score ranked at the 9th percentile.
The same outcome, and even higher relative ranks because of the reduced percentiles, was observed for each of the other 4 laboratories.
To summarize the clinical utility of tool customization, it is possible to bypass the general tool available to all sites. This flexibility is needed to factor in the great variability of analyte selection adopted by different laboratories. In most cases a site-specific tool is a “scaled down” version that is still sufficient to detect most cases affected with the target condition. However, the possibility of a lower sensitivity should be carefully considered. Finally, tool customization is rapid and easy and could be performed by any user with access to the tool builder. Many sites have taken advantage of the opportunity to create their own customized tools, the only requirement to be met is to be up to date with the submission of all types of data within the scope of R4S: participant profile, reference percentiles, true positive cases, and performance metrics. Because of the very nature of the post-analytical tools, cutoff values are optional as they have become increasingly irrelevant to the interpretation process.
This is the conclusion of part V of the R4S series of Mayo Medical Laboratories Hot Topics. In part VI we will provide an overview of the status of other newborn screening applications within R4S and describe the most significant improvements and new features of the upcoming version 2.0 of the CLIR software, to be release between late 2013 and first quarter 2014. As a reminder, CLIR stands for Collaborative Laboratory Integrated Reports. We believe CLIR is applicable to a much broader analytical landscape beyond newborn screening, one that covers the entire field of Clinical Biochemical Genetics and potentially many other clinical but also research areas of laboratory medicine and pathology.
Please do not hesitate to contact us if you have any questions or requests related to the content of this presentation. Thank you very much for your attention.
Piero Rinaldo, M.D., Ph.D.
Professor of Laboratory Medicine and Pathology,
Mayo Clinic, Rochester, Minnesota
Thank you for the introduction. This presentation is the final segment of a 6 part series describing the products and clinical tools of a newborn screening quality improvement project called Region 4 Stork, or R4S.
I have a disclosure to make: a provisional patent application related to the content of this presentation has been submitted by Mayo Clinic. The title of the application is “Computer-Based Dynamic Data Analysis.”
The title of this presentation is “Other applications and future developments.”
Although the MS/MS application has been the primary focus of the entire series, there are 8 more live applications on the R4S website. They are devoted to other established targets of the recommended uniform screening panel but also to some new tests under development and validation. The list of established applications includes congenital adrenal hyperplasia, biotinidase deficiency, routine second sample analyzed by tandem mass spectrometry, shown as MS/MS [2], and severe combined immunodeficiency, or SCID. The applications related to new tests and conditions are Lysosomal storage diseases, Friedreich ataxia, x-linked adrenoleukodystrophy, and other Peroxisomal disorders, and finally a combined application titled Pilot study that includes the data of the 3 previous applications and one more condition, Wilson disease.
A defining characteristic of all these applications is the consistency of format and content, namely all the first generation and post-analytical interpretive tools. For example, the plot by marker for T-cell recombinant excision circle, the analyte measured to screen for severe combined immunodeficiencies that is abbreviated as TREC, looks exactly like the tool in the MS/MS application showing the cumulative reference range and, in this figure, the disease ranges of 19 separate conditions. Likewise, the all conditions tool for 9 lysosomal storage diseases is identical to the one found in the MS/MS application. The relevant point to be made here is that familiarity of a user with one application allows the utilization of all the others.
This table summarizes the status of the 9 R4S applications as March 31, 2013. The MS/MS application is approaching the mark of 15,000 true positive cases and has surpassed the milestone of 1 million data points. The other modules include between 77 and 866 true positive cases, on average more than 300 per application. Even if significant smaller, these are volumes that a single laboratory would be hardly pressed to assemble in isolation without cooperation.
R4S constitutes a novel approach to the interpretation of laboratory test results, one that has the potential to create net value in the practice of both the performing laboratory and the ordering physician. R4S has reached an unprecedented level of worldwide collaboration, 185 laboratories in 51 countries to date, and has evolved into a testing environment for a continuous and dynamic clinical validation process. Clinical significance is not based on arbitrary choices but is entirely evidence-based. Reference ranges are not expressed as greater than or less than but as cumulative percentiles. The same is true for disease ranges which are condition specific, not cumulative by analyte. Cutoff values are simply not necessary. Peer comparison is extensive, transparent, available on demand and always up to date. The selection of ratios is facilitated by several tools and plot for data analysis. Like in the case of cutoff values, sequential algorithms are made obsolete, and replaced by the parallel, simultaneous evaluation of all informative markers. Differential diagnosis is automatic, and in most cases resolved by dedicated dual scatter plots. Subjectivity is minimal, and shared knowledge is practically built in. Rather than resting on these accomplishments, this is exactly the time to ask, “What is NEXT?”
R4S is likely to continue to evolve, starting with additional newborn screening modules. In the near future a new application will be launched to collect data and create tools based on the first round of confirmatory testing once a referral has been made. This application will be called “Newborn screening short term confirmatory testing” and will collect the data of the traditional tests performed in plasma and urine for confirmation purposes. When applicable, each condition will be split in 2 groups, true positives and false positives to allow the creation of dual scatter plots targeting the differential diagnosis between them.
In addition to new R4S applications, a new cluster called CLIR 1.0 has been created. As described in the next slides, they cover other tests in Clinical Biochemical Genetics, other specialties of Laboratory Medicine and Pathology, and some basic research applications. Later in the presentation, version 2.0 of the software will be briefly introduced.
This effort is driven by the belief that R4S tools are NOT just applicable to newborn screening, but they can provide useful answers to 3 basic questions in a broad spectrum of clinical circumstances: detection of an overall profile that fits the pattern of a target condition (yes or no answer), a differential diagnosis between 2 conditions with similar and overlapping biochemical phenotypes, and the recognition of the most likely condition among a large number of possible choices.
As a reminder, the IT infrastructure of R4S was initially located at the Michigan Public Health Institute in Lansing, Michigan. In the summer of 2012, the R4S website and applications became part of the Newborn Screening Translational Research Ne2rk based in Bethesda, Maryland. At the same time, a new and completely separated cluster of applications was created within the Mayo Clinic IT infrastructure. This new project has been named Collaborative Laboratory Integrated Reports, or CLIR 1.0.
CLIR 1.0 applications target any test performed by the Biochemical Genetics Laboratory (BGL for brevity) that rely on selection of cutoff values and pattern recognition of complex metabolic profiles. To date, approximately 30 such applications have been created on a Mayo intranet development site. A screen shot of the log in page is shown here.
Just like in the case of newborn screening, the availability of very large numbers of true positive cases is critically important to establish condition specific disease ranges for all the analytes of interest. If we consider the current size of the R4S database as a goal to emulate, between 1500 and 2000 cases per year are needed over a period of 10 years.
Fortunately, data archiving of true positive cases has been a priority of BGL since 1999, and indeed a similar if not greater number of cases have been collected during the past 10 calendar years.
This table shows the 11 major groups on inborn errors of metabolism in the BGL database, and the number of diagnosed cases per category per year. Overall, more than 20,000 cases are already available to create disease ranges for a broad number of inherited conditions.
This table shows the status of the 10 BGL applications with the highest number of true positive cases with a legend of the abbreviations listed in the header line. 2 of them, urine acylglycines (A-C-Y-L-G) and plasma very long chain fatty acids (POX) will be highlighted in the next slides. Between 100 and 1200 cases per application have been already uploaded to CLIR 1.0 applications. However, this work has just started considering that only16% of the total number of available cases has been processed so far.
This effort is driven by the expectation of achieving a number of clinically useful deliverables: first, the automated production by any instrument software in the laboratory of a .csv file inclusive of fully anonymized batches of clinical sample raw data. A second deliverable is the creation of a post-analytical interpretive tool for every condition potentially detectable by a test or a combination of multiple tests. The third and final goal is the routine processing of such data batches by the tool runner in every CLIR application. Obviously, the sheer magnitude of this effort is justified if it can be proven that there is added value to be found in this process.
The first example of added value is provided by the plasma very long chain fatty acids test. This test is primarily used to recognize several inherited disorders affecting the function of the intracellular organelles called peroxisomes. This test requires a complex differential diagnosis between single enzyme defects and global disorders of peroxisomal biogenesis. By reproducing the process described for newborn screening, the frequent occurrence of minimal elevations of one or more measured analytes, findings that often trigger a request for a repeat sample, can be replaced by post-analytical tools that conclusively override those random and non-specific results. The improved specificity is not the only added value: preliminary workload recording has shown a reduction of average review time by a laboratory director from 2 minutes to 12 seconds per test, a reduction of almost 99%.
The same evaluation is under way for the urine acylglycine profile, another high-volume test where the utilization of the tool runner in routine daily practice is ready to be activated.
It should be apparent that CLIR could be used for any other multi-analyte test performed by a wide spectrum of clinical laboratories that require expert interpretation of complex profiles. Notably, the test catalog of the Department of Laboratory Medicine and Pathology at Mayo Clinic includes at least 463 tests with more than one reportable result, more than half of them actually include 5 or more markers.
Finally, CLIR could support the data collection, secure web-based sharing, and comparison between sites collaborating on research projects that include the characterization of biochemical phenotypes. To date, a dozen applications have been started already, new ones could be added almost instantly when a request is brought to our attention.
Requests to activate a new CLIR application, either clinical or research, have relatively simple requirements to be fulfilled, a list we call the “starter kit”. An application needs one or more content experts who are ready to assume the role of curator of the database. These individuals are given administrative access to the full arsenal of tools, including the tool builder. To populate the log-in icon a short and long name are needed (for example, POX and plasma very long chain fatty acids). A description of the condition types and again short, long name, and, if applicable, SNOMED code of the individual conditions which are the targets of the application. Analytes can be sorted in categories called “types”, again full descriptions and preferred abbreviations need to be provided. The individual analytes should also describe the unit of measurement and, if available, the LOINC code on record that matches the unit and the specimen type. Once the framework of conditions and analytes has been set up, a process that could be completed very quickly for an application of average complexity using the administrative tools available on the website, the next step is to calculate and upload analyte percentile values of reference subjects. The final step is to enter one by one all available true positive cases. This process can be facilitated by semi-automated uploading of .csv files that can be easily generated from available Excel spreadsheets. Post-analytical tools could be activated with data from as little as 5 cases, obviously much larger numbers are needed for a tool used to analyze prospective data in a clinical setting.
A frequently asked question is if there is a limit to the number of analytes that could be added to a single application.
While there is no set limit in the software, a first attempt to test the reliability and speed of the tools when large numbers of analytes need to be processed was tested using an application for Cystic Fibrosis mutation screening based on a mass spectrometric method. This model fits well the 3 clinical questions described earlier, having to answer the question if a given case is a CF carrier or not, if the case is either a carrier or is affected, and having to pick one or more mutations from a total of 106 different alleles.
As shown on the left side of this slide, one single case amounts to 434 analytes. Each mutation is defined by a measure of the signal to noise ratio and by the peak height of the wild type and mutant signals as shown in this partial enlargement. Not shown here, these results are applied to calculate a total of 3776 ratios. It is therefore legitimate to ask if the tool runner could process this amount of data on a routine basis.
Including controls, one routine batch includes 48 specimens, and more than 20,000 results. CF mutation screening is a high-volume test that requires the processing of >1000 batches per year and >20 million results, not including the calculated ratios.
The tool runner was not challenged by this load and in only a few seconds can process a batch. Shown here is the all conditions tool for a case who is a carrier for the most common CF mutation, DF508.
This observation is promising but it should be mention that there are other limitations of the R4S/CLIR software that still need to be resolved. Incomplete sets of data interfere with the calculation of ratios, negative values cannot be processed, and there are instances where tests results are expressed only as a binary choice, positive or negative. Another frequent problem is the lack of measurable results (when normal is equal to zero) to calculate reference ranges. Finally, values less than 1 and especially less than 0.1 should include a sufficient number of significant decimal digits to avoid artifactual clustering in the data display tools.
These are just some of the reasons behind the decision to develop a second generation of the software that has been named CLIR 2.0.
This slide shows the current appearance of the plot by condition in 1.0 and a prototype of the same tool in CLIR 2.0. In addition to a sharper graphic definition, users will actually be able to choose a color-palette if they prefer not to use the default colors, mostly red and green, which are used in the tools.
This slide is quite crowded and yet it is only a partial list of the large number of improvements and new features that will be incorporated in the new version of the software. One that is worth mentioning is shown at the top of the list: in 2.0 it will be possible to generate on demand dual scatter plots for any 2 conditions a user is interested to compare in a given case. Currently, this tool requires extensive preliminary work behind the scenes to set up a pair of matching 2-conditions tools, and their merging in a dedicated dual scatter plot that also requires editing before it is released into production.
Beside improvements and new features, the most compelling reason to develop a new version of the software is the realization that 1.0 had become some sort of an inverted pyramid: it started as a single application (MS/MS), then several other applications were added on top of the first one, all similar but with unique requirements that prompted a number of customized changes, and now potentially hundreds of CLIR applications are being developed. This progression is not sustainable without running the risk of making the whole infrastructure unstable, and prone to outages. This is why version 2.0 is needed to create a broad and robust foundation able to support the workload and diversification of both R4S and CLIR.
In parallel to the coding of the new infrastructure, work will continue to explore the possibility of creating a CLIR application for every test we do, and to find the dedicated content experts who will assume the oversight and curation of data collection to establish reference range and condition-specific disease range percentiles, and the transfer of archived data to the appropriate CLIR application. At the same time, it is necessary to establish and implement a routine process to capture more data prospectively. This is the basis of the concept described earlier of constantly evolving, dynamic clinical validation. As the number of applications grows, it will become a necessity to train a larger group of super-users who are proficient in the use of the tool builder. Indeed, the ultimate goal is to create post-analytical tools for every target condition of a laboratory test.
The pursuit of this arguably ambitious plan could lead us to significant outcomes: objective and quantifiable improvement of test performance, utilization, and ultimately of patient care. As already mentioned, reliance on a quality assurance system based on constant, not static clinical validation. Tools could improve the consistency of interpretation among multiple laboratory directors but also physicians who alternate in covering either a section of the laboratory or a clinical service. Education will also benefit, as students, residents and fellows will have immediate access to large bodies of objective evidence that will remain available to them once their training is completed and they move along in their professional career. It is likely this type work will translate in greater academic visibility and productivity. CLIR applications could also offer an opportunity to all laboratory personnel, not just physicians and scientists, to learn new information management skills and to foster enduring professional satisfaction. These outcomes combined could lead to improved performance and also expense reductions, for example by facilitating a systematic conversion to a paperless process. Last but certainly not least, the most far-reaching goal of version 2.0 of the CLIR software is to obsolete the need to establish age-matched reference ranges for test results particularly in a pediatric population.
This work is the scope of an exciting collaboration we have established recently with 2 investigators from Oslo University Hospital, Drs. Lars Mørkrid and Alexander Rowe. They proposed to replace our static cumulative percentiles with the collection of data points that are converted to z-scores, an independent measure of deviation that has been illustrated in a previous segment of this series and applied over an age continuum. This process also allows a statistical robust exclusion of outliers, when applicable.
The expected end product of this new line of research is the creation at the front end of the CLIR software of the equivalent of “growth charts” for every analyte under consideration, and the expression of a patient result, shown here for a patient between 1 and 10 years of age as a white circle between the 90% and the 97.5% percentiles, in a manner that is free of arbitrary age “bins” and, like R4S and CLIR, free of equally arbitrary cutoff values.
This animated slide should lead to a better appreciation of the evolution of the R4S/CLIR software. The starting point is the status quo, age-matched reference ranges and arbitrary cutoff values. The first step was the systematic adoption of reference percentiles and condition specific disease ranges, also calculated as percentiles. Once these ranges were established with adequate power, it was possible to obsolete the conventional approach and define cutoff target ranges. The limitations of the target ranges, and a less than ideal overall utilization, lead us to focus on the degree of overlap between reference and disease ranges, overlap that could be anywhere between substantial and not existent. Although very effective in newborn screening practice, this approach is not corrected for age and needs to be once again replaced this time by the definition of z-score percentiles.
These slides show how the current software would be forced to look at patients with the same condition but clustered according to age. Moreover, in the first generation of interpretive tools the focus of data analysis had been on the range LIMITS, and the consequent estimate of the degree of overlap.
In the next generation of tools, the focus will be placed on the degree of dispersion of the data within a single continuous age range. In addition to the significant advantage of eliminating the need to establish separate applications for different age ranges, preliminary evidence has shown that reference and disease percentile ranges will be segregated more effectively, leading to more sensitive and specific tools.
Age is the most obvious but just one of the covariates that the 2.0 software will be able to incorporate. This figure shows again the work of our colleagues from Oslo University Hospital who were granted retrospective access to approximately 90,000 newborn screening results for the marker 17-OH progesterone. Correction for 2 additional covariates, birth weight and gender, in addition to age at collection, resulted in a very promising distribution of values that could lead to the selection of a single threshold thought the continuum of birth weight values, the root cause of historically poor performance of newborn screening for congenital adrenal hyperplasia.
In summary, CLIR 2.0 will be capable to provide front end correction of laboratory results for multiple covariates, will continue to foster worldwide collaboration and data sharing, and will enhance existing high throughput data portals for batch data submission to the tool runner.
In the end, a virtually unlimited number of web-based, always up-to-date, and on-demand post-analytical tools will become available to a broad spectrum of users in clinical but also research practices.
This is the conclusion of part VI of the R4S series of Mayo Medical Laboratories Hot Topics, and also of the entire series.
I would like to acknowledge several individuals who have contributed to the work presented in this series, beginning with the team of programmers and code developers, particularly David McHugh and Gregg Marquardt. The importance of the contributions by our Norwegian colleagues Lars Mørkrid and Alex Rowe cannot be overstated. I also want to recognize my BGL colleagues, laboratory directors, current fellows and genetic counselors who have spent countless hours populating and testing applications as they became available. Many other individuals in BGL have also contributed, just too many to mention here.
Finally, I would like to thank the MCSI scientific and technical publication team lead by Denise Masoner for their outstanding support, and patience, during the recording of this series.
Please do not hesitate to contact us if you have any questions or requests related to the content of this presentation. Of course, we will be happy to provide a password to R4S to any interested new users, just send a request to the email address RSTCLIRSupport@mayo.edu. Thank you very much for your attention.
Contact us: mcleducation@mayo.edu
