443-287-4832

Menu
Quality Assurance for Assays and Interpretation of Results

Introduction

This module describes the process of monitoring quality assurance of assay testing and interpreting results from serosurveys.

Topics covered in this module

  • Monitoring quality of assay testing procedures
  • Common issues with assay procedures
  • Interpretation of results from EIAs

Note: Refer to below modules for related information

  • Specimen collection procedures and transport from the field to the laboratory module
  • Specimen processing, storage, and shipment module(link)
  • Developing and running assay protocols module

Quality Assurance Considerations

Overview of Quality Assurance

Throughout the course of the enzyme immunoassay (EIA) protocol, quality measures related to the assay should be implemented to ensure the precision, reliability and validity of the results.

Precision: How similar are two or more independent EIA measurements of a specimen to one another?
Accuracy: How well does the EIA determine the true level of antibodies in the serum specimen?
Reliability: How consistent are the EIA results when repeated, or the reproducibility of the EIA?
Validity: Is the assay actually measuring antibodies to an antigen?
  • Calibration protocols using a reference IgG antibody standard should be performed to ensure accurate and precise results. This should be done at the beginning of testing and if testing survey specimens with different lots of the same assay kit.
    • This can be done with a known panel developed internally or using WHO International standards.
    • Commercially available EIAs contain calibration instructions that should be followed at the beginning of testing and if using different plate readers. These protocols will need to be developed by the laboratory if using an “in house” EIA.
  • With even the most reliable tests and most rigorously developed lab protocols, the imperfect nature of EIAs means some level of variability and discordance will occur with retesting protocols. To deal with these, the following re-testing protocols are recommended to ensure the reliability of the serosurvey lab protocol.
  • Establish a regular retesting protocol of random or fixed interval specimens on each plate to confirm the protocol is consistent throughout testing.
  • Establish a protocol for handling discordant results. (See Interpreting EIA results)

Goal

Monitoring Assay Quality

Routine
Retesting for Quality Assurance

Type of specimen or reference used

Manufacturer-provided calibrators
and controls

Internal controls

Random retesting of serosurvey specimens

Description

Reference values provided by manufacturer to check assay
validity

Developed in-house or using International Reference Standards

 

Between plate (inter) and within plate (intra) assay correlation of specimens tested more than once

How evaluated

Tracking values over time or comparing between laboratories or technicians

 

Calculate the following using the quantitative values

        Coefficient of variation

        Correlation coefficient (R2)

Table 1. Summary of different methods for quality assurance


Monitoring Assay Quality

  • It is important to assess quality of the assay over time. Options for external validity assessment include running international standards, a known serum panel, or a validation subset with another EIA or gold standard assay.
  • Commercially available EIAs should contain calibrators and controls that have been validated against a reference standard. These are tested on every plate to ensure that the results are valid.
    • The reference values should be included with each test kit. The values for the calibrators and controls must lie within the limits stated for the kit.
    • If the values specified for the calibrators and controls are not achieved, the test results may be inaccurate and testing should be repeated.
    • To allow for rapid identification of problems, recommend adding assay validity checks to the EIA Results template to check the observed values against the reference values and to flag when the specified values are not achieved. Refer to Figure 2 and the “CALIBRATION_ANALYST” tab in the tool, EIA TEMPLATE BLANK, for an example.

Figure 2. In the “Calibration” tab of the template the analyst can record the Optical Density (OD) values for the calibrators and controls (PC positive control; NC, negative control) in the blue cells. The template automatically calculates the standard curve and determines if the assay was valid or not based on the values of the calibrators and controls.

  • Tracking calibrators and negative and positive controls
    • Another way to assess the quality of EIAs is to track the values for calibrators and controls of multiple kits from the same lot.
    • This allows visualization of trends, such as decreasing calibrator values or outliers if one plate’s values were higher or lower than other plates.
      • If using multiple laboratories for testing or multiple laboratory technicians, can also compare these data across laboratories or technicians.

Figure 3. Monitoring Optical Density (OD) values for calibrators and controls of the same lot across plates to identify potential problems with assay runs. Although some plate-to-plate variation is normal and expected, the flatter the line, the more uniform the values of the calibrators and controls over time.

  • Using internal controls
    • Another way to monitor the quality of specimen testing is to use an internal control on every plate.
      • This could be obtained from a blood bank or some other biorepository. It should be a well-characterized specimen with known antibody concentration and sufficient volume to be used on all EIA plates run for the serosurvey.
      • This could also be done using the International reference standards.
    • Since this will always be the same specimen, monitoring the antibody concentration obtained from every plate allows you to assess if the value is changing over time or across technicians or laboratories.
      • For example, if there is an outlier value, that plate should be reviewed.

Routine Retesting for Quality Assurance

  • The goal of routine retesting is to systematically assess intra-plate (within the same plate) and inter-plate (on different plates) variability.
    • This provides a sense of how accurate and reliable the assay results are.
    • Can choose to do as many retests as desired but may want to keep <10% to remain manageable. Can also choose to start by doing more retests for quality assurance and decrease overtime if the serosurvey will be testing large numbers of specimens and there is no evidence of problems with assay quality or testing performance.

Intra-plate (within plate) variability can be assessed by randomly repeating a few specimens on the same plate.

  • Can be done by systematically selecting every nth specimen to be retested at the end of the plate, as seen in the blue cells in Figure 4, and assess if there is a difference between the original and the retest value.
    • Ex. Calculate the coefficient of variation between the first and second values.
    • Ex. Calculate the correlation coefficient between the first and second values.
  • Data resulting from intra-assay validation helps ensure that specimens run in different wells of the plate will give comparable results.

Inter-plate (between plate) variability can be assessed by randomly repeating a few specimens on a different plate.

  • Can be done by systematically selecting every nth specimen to be retested on a retest plate, as seen in the green cells in Figure 4, and assess if there is a difference between the original and the retest value.
    • Ex. Calculate the coefficient of variation between the value on the first plate and value on the second plate.
    • Ex. Calculate the correlation coefficient between the value on the first plate and value on the second plate.
  • Data resulting from inter-assay validation helps ensure the results are consistent over time and between plates.
  • Typically, a retest plate will include all specimens to be retested, including retests for quality assurance, equivocals, and specimens above the upper limit of detection (diluted for retesting).

Coefficient of variation is calculated by dividing the standard deviation (σ) of a set of measurements by the mean (µ) of the set of measurements and multiplying by 100 to get %.
Cv=σ/μ*100
This calculation could be done for the optical density (OD) values or the antibody concentration calculations. The smaller the coefficient of variation, the more reliable the estimate. One rule of thumb guidance is <10% acceptable, 10-14% questionable, >15% unacceptable and should be reviewed.

Correlation coefficient (R2) denotes the relationship between the two measurement values. The closer the R2 is to 1, the more tightly correlated the two measurement values are. This can be visually seen on a scatterplot or calculated using a data program (e.g. Excel, STATA, R).

Figure 4. Example plate map demonstrating how specimens to be retested can be selected. Yellow cells indicate calibrators and controls provided in the EIA kit as well as in-house internal controls. Blue cells indicate specimens retested for intra-plate quality assurance. Green cells indicate specimens to be retested on another plate for inter-plate quality assurance.

Figure 5. Example scatterplot evaluating intra-assay correlation. The x-axis is the quantitative value for the first well and the y-axis is the value for the second well on the same plate. The red line indicates equivalence. The light gray lines indicate equivocal and positive thresholds. The point circled in red reflects a specimen that was categorized as equivocal in the first well and categorized as positive in the second well (using 200 mIU/mL as the positive threshold). It is important to focus on values around these thresholds since minor quantitative changes may result in qualitative differences in terms of seropositivity.


Common Problems with Assay Procedures

Monitoring specimen testing can help identify problems with assay procedures. Refer to Table 1 for examples of common problems.

Issue How to identify
Duration of time from placing first specimen on the plate to the last specimen on the plate is long. Intra-plate retesting demonstrates systematically lower values for the repeat tests at the end of the plate than in their original well (Figure 4).
Clog in the plate washer or one loose-fitting pipette in a multi-channel pipette. Values along the same row of a plate are consistently high or low, particularly if comparison is of repeated testing values.
One laboratory technician is not carefully following the assay protocol. Monitoring the inter-plate retesting, intra-plate retesting, and values of the calibrators and controls by a laboratory technician can help pinpoint whether there is more variability seen with a particular technician’s plate runs.
Too much time elapsed during one step of the EIA. Laboratory technicians should report if the timing at any step of the EIA exceeds that allotted by the kit instructions. Another way to identify this is if the values for that plate are systematically higher than the values of the repeat tests that are done on a retest plate.
Specimens have degraded from too many freeze-thaw cycles. If possible, specimens should only be thawed once for testing and any retests are done quickly thereafter. If the retested results are always found to have lower antibody concentrations than the original values, this could indicate that specimens are degrading.Toolkit-Data Procedure EIA Assay Template Example
Differences in quality between lots of EIAs.

A lot is a set of multiple EIA testing kits that were all calibrated and validated together, so they have the same reference values.

While there is some inherent variability expected between lots, they should all be able to provide valid results. When starting a new lot, consider retesting a subset of specimens already tested from the previous lot to assess concordance. If using internal controls, compare the antibody concentrations from plates that used different lots.

Note: the OD values will differ between kits, but the antibody concentrations should not.

Table 1. Common specimen testing issues


Interpreting EIA Results

Assign outcome value to results

Decisions about assigning final outcome values should be specified clearly in the study protocol and adhered to throughout the analysis to ensure validity in final prevalence calculations.

  • If specimens are tested only once, use that result as the final value assigned. If specimens are tested more than once, establish a protocol for concordant and discordant results.
  • Concordant seropositive or seronegative: Assign agreed upon value.
  • Discordant:
    • The study protocol should clearly specify which result to use for a specimen in the analysis
      • Ex. 1: Use the first available result.
      • Ex. 2: Replace initial equivocal result with repeat result if the specimen was positive or negative on re-testing.
  • Concordant, but equivocal:
    • The study protocol needs to clearly guide how these outcomes should be categorized in prevalence analysis. Three possible strategies:
      • Include equivocal results as seropositive results
        • Biologically plausible: The EIA threshold for a positive result is set at a level of IgG that is higher than the minimum level required for detection.
      • Include equivocal results as seronegative results
        • Conservative approach that will lead to some underestimation of seroprevalence, which may be preferred to overestimation.
      • Keep equivocal results as a separate category
        • Permits a reviewer to conduct a sensitivity analysis, if desired, by combining the equivocal results with either the positive or negatives.
  • Refer to the module, Developing assay protocols, for more information on handling equivocal specimens.
  • Note: The decisions regarding recategorizing equivocal results and the numbers of recategorized specimens should be reported clearly in reports and publications such that readers can perform a sensitivity analysis based on these results if desired.
  • WHO Serosurvey Manual chapter 4, Section 4.5 Interpreting EIA Results

Figure 6. Example of summary flowchart to assign qualitative values to tested specimens

Determining Immunity

Immunity is a complex process, and IgG antibody detection by EIA provides only a surrogate measure of immunity.

  • While assay cut-offs were established to maximize test specificity, most assays were designed to evaluate an individual’s immune response and not for population antibody prevalence estimation.
  • A negative or equivocal assay result is typically interpreted as susceptibility to disease, but this may not necessarily be the case.
    • Imperfect sensitivity may be a concern with serum IgG EIAs, meaning that the assay may misclassify someone as seronegative when in fact they are seropositive.
    • An equivocal IgG EIA results may meet threshold cutoff for immunity by neutralization assays (e.g., Plaque reduction neutralization assays or PRN found in the module, Developing and Running Assay Protocols Module).
    • Circulating antibody levels may be lower after vaccination than those occurring after infection. Therefore, low level antibodies resulting in equivocal results would be expected with commercial EIA assay.

Some antigens have established thresholds for protection such that immunity can be thought to provide protection.

Some assays may not have established cut-offs to determine immunity, requiring more complex methodologies for interpretation such as mixture modeling1.


References:

World Health Organization Guidelines on the Use of Serosurveys in Support of Measles and Rubella Elimination, Chapter 4 Laboratory Methods

World Health Organization Manual for the Laboratory-based Surveillance of Measles, Rubella, and Congenital Rubella Syndrome

  • Chapter 9. Laboratory testing for determination of population immune status

International Reference Standards

1. Vyse, Andrew J., N. J. Gay, L. M. Hesketh, R. Pebody, P. Morgan-Capner, and E. Miller. 2006. “Interpreting Serological Surveys Using Mixture Models: The Seroepidemiology of Measles, Mumps and Rubella in England and Wales at the Beginning of the 21st Century.” Epidemiology and Infection 134 (6): 1303–12. https://doi.org/10.1017/S0950268806006340.


Toolkit

Section Toolkit material Context
Monitoring quality – assay validation Toolkit-Data Procedure EIA Assay Template Blank Template for use with Euroimmun measles or rubella EIA kits
Monitoring quality – assay validation Toolkit-Data Procedure EIA Assay Template Example Template for use with Euroimmun measles or rubella EIA kit
Monitoring quality – retesting Toolkit-Example Protocol for Assay Retesting Protocol developed for use in a measles and rubella serosurvey