Why System Suitability Tests Are Not A Substitute for Analytical Instrument Qualification or Calibration (Part 2)

Part 1 of this article in the July/August issue looked at the regulations and quality standards around system suitability tests (SSTs) and discussed the activities in the data quality triangle. The second part will discuss why SSTs cannot be used for analytical instrument qualification in more detail.

In the previous article on system suitablity tests¹ we looked at the data quality triangle from United Stated Pharmacopoeia (USP) <1058> on analytical instrument qualification (AIQ)² that consisted of the four interlinked layers shown in Figure 1 that defines analytical data quality on the basis of the correct application of them:

• Analytical instrument qualification (AIQ).

• Analytical method validation.

• System suitability tests.

• Quality control (QC) check samples.

Regardless of the source of the quality triangle as a regulation, this approach is applicable to all chromatography laboratories whether they operate in a regulated or quality environment, as it describes good analytical science as well as good business by protecting the investment. In Part 1 I modified the triangle to identify:

• Is an activity instrument or method dependent?

• When is the activity performed?

• What does the activity control?

The Foundation: Analytical Instrument Qualification

USP <1058> describes analytical instrument qualification as the process for ensuring that an instrument is suitable for its intended purpose as we discussed in the earlier column.³ This is the lowest level of the data quality triangle shown in Figure 1 and is the foundation for all other stages of analytical work. <1058> defines analytical instrument qualification as "the collection of documented evidence that an instrument performs suitably for its intended purpose".² The end result of the AIQ process answers the question: Do you have the right system for the right job?

Figure 1: The data quality triangle modified from USP .

The AIQ process is broken down into four phases, known as the 4Qs model:

• Design qualification (DQ) is used to define the user's requirements before purchasing an instrument or system.

• Installation qualification (IQ) demonstrates that the components have been correctly installed.

• Operational qualification (OQ) shows that the installed system meets the user specification. A periodic OQ is also performed after major maintenance or service of an instrument.

• Performance qualification (PQ) demonstrates that the system continues to perform as defined.

Note that the term performance verification (PV) is not used in USP <1058> or this document, as performance qualification (PQ) is the preferred term.

What happens if you don't qualify your equipment? As Avalone in Crowther and Miller⁴ notes, "One area that is frequently cited in FDA 483 observation reports is the failure to calibrate and maintain laboratory equipment" and this is borne out below in just three of many warning letters for the laboratory:

ChemSource warning letter from November 2002:⁵

The inspection revealed that your laboratory equipment calibration program is inadequate in the following ways:

(a) Failure to have written procedures describing specific calibration instructions and limits.

(b) Failure to maintain complete calibration records in that they do not include all raw data.

(c) Failure to have a complete calibration program for the HPLCs in that the gradient accuracy and detector linearity are not being verified.

(d) Failure to conform to the USP <41> for weight and balance determination. The inspection revealed that erroneous values are being used to perform the minimum weight studies.

Earlham College warning letter from July 2002:⁶

(3) Failure to comply with the General Requirements of Subpart I. Laboratory Controls, as required by 21 CFR 211.160, in that there is:

(f) No established written program for the maintenance and calibration of instruments such as the atomic absorption and HPLC instruments and the [redacted] balance used for drug analysis.

(g) No certification to a recognized standard for the weights set used for checking the [redacted] balance.

Ion Laboratories Warning letter from July 2009:⁷

Your firm has not conducted adequate calibration of instruments, apparatus, gauges, and recording devices at suitable intervals in accordance with an established written program containing limits for accuracy and precision [21 CFR § 211.160(b)(4)]. For example,

(a) Your firm failed to conduct injector and detector performance testing for the HPLC system. For example, no HPLC injector and detector testing for linearity, accuracy, and precision were conducted, such as:

(1) Various injection volumes and standard concentration testing;

(2) Evaluation of detector for noise/drift; and

(3) Carry over testing.

It is important to note the type of HPLC qualification testing required in these warning letter examples. Gradient pump accuracy, autosampler injection volume accuracy and precision and detector linearity are just some of the tests that are expected to be performed. These parameters are not checked explicitly versus external test equipment and standards in a system suitability test that focuses on method-specific parameters such as retention time windows, peak shape and resolution between peaks of interest as well as column performance. I would also reiterate the need for equipment to be calibrated against national or international standards whenever possible — hence the citation of Earlham College for failing to use weights calibrated to a recognised standard.

Table 1: Analytical instrument qualification questions and issues to consider.

The message here is very clear — ensure that the instrument or system has been qualified and/or calibrated as necessary because it is the foundation of all further analytical work. Failure to adequately qualify may result in poor quality analytical results and will give problems when transferring analytical methods to other laboratories. Many of us have experienced method transfer projects that failed their acceptance criteria due to variance in results directly attributable to performance differences between the instruments used in the originating and establishing laboratories.

Table 2: Analytical method validation questions and issues to consider.

This is the only part of the data quality triangle that is instrument specific: there is no other check of the fundamental operation of the instrument or system. All remaining phases of the triangle are method specific, showing how important this foundation layer is.

Analytical Method Validation

The next stage up the data quality triangle is analytical method validation and this, not surprisingly, is method based. Method validation relies on the fact that the instrument is qualified and works correctly; the corollary is that methods developed and validated on unqualified equipment can raise questions over the quality of data generated. The overall method variance may be higher and may have less robustness compared with data generated from qualified instruments.

Table 3: System suitability test questions and issues to consider.

USP <1058> defines analytical method validation as "The process to demonstrate via documentation that any analytical procedure is suitable for its intended use".² Again a simple interpretation of this statement is: Have we got the right method for the right job?

However, before starting work on validating the method, the AIQ data quality triangle implicitly assumes that the instrument/system is qualified and therefore gives the right results against the laboratory specifications. Therefore, if an instrument has not been qualified and further work in the higher levels of the data quality triangle is suspect. Thus if you are assured that the instrument gives the right results a method can be developed and validated on it.

Table 4: Quality control check questions and issues to consider.

System Suitability Test

The third layer of the data quality triangle is the system suitability test. Again the basis for an SST working reliably is that the instrument is qualified and the method used is validated. USP <1058> defines this as: "Verify that the system will perform in accordance with the criteria set forth in the procedure".² What this really means is: Is the method running on the system working as you expect on the day you want to analyse samples and before the samples are committed for analysis? This approach is good analytical science and this should be the driver to perform adequate system suitability tests rather than simply compliance with regulations.

Quality Control Checks

The apex of the data quality triangle is quality control checks. Again, these are method-based checks to ensure that the analytical system works throughout the run within predefined limits. Typically these are independently prepared samples of known concentration or amount analysed as unknown samples and are calculated as such to confirm that the instrument and method works correctly. They are typically interspersed throughout the run to provide an additional confirmation that the chromatographic system is continuing to work.

Summary of the Components of Data Quality

Although cited from a regulatory source, the components of data quality shown in Figure 1 should not just be looked at from the perspective of a regulatory requirement. This triangle is fundamental to good analytical science and also provides substantial business benefit because the approach reduces the risk of releasing inadequate analytical results. The performance of the instrument is established and monitored to see that it delivers quality results.

Some of the undocumented key points of the data quality triangle are as follows:

• AIQ and method validation should be completed before the analysis takes place and not after because there is an implicit time scale from the bottom to the top.

• SST is performed immediately before the analysis — this is the first on the day check to see that the instrument running the method works within set limits.

• QC sample analysis is performed throughout the analytical run — this is the second on the day check to ensure that the instrument and method performance does not change over time.

Furthermore if you don't use this data quality triangle you run the real risk of:

• Buying an expensive inappropriate or unsuitable instrument;

• Using the instrument in an invalid way. For example, outside of an accepted operating range; or

• Making unsuitable decisions from the data generated by the instrument.

Why SSTs Are Not AIQ

Let us return to the subject of this column and look at why a system suitability test cannot replace analytical instrument qualification. We will focus on the instrument function tests performed during the operational qualification (OQ) phase of the AIQ. A typical argument about using SSTs in place of AIQ goes something like this: "Our laboratory does not need to qualify the instrument because we run SST samples and they are within limits".

There are a number of problems and fallacies with this argument, but rather than make my own arguments about why this is so, I will use quotations from the USP <1058> analytical instrument qualification process section to do this on my behalf and then I'll provide additional comments.

Operational Qualification and Instrument Function Tests

Instrument functions required by the user should be tested to verify that the instrument operates as intended by the manufacturer. Users, or their qualified designees, should perform these tests to verify that the instrument meets manufacturer or user specifications in the user's environment.²

In essence, this means that the functions of an instrument must be tested. Using a liquid chromatograph as an example, the flow rate of the pump (maximum and minimum flow rates as a minimum) and the wavelength accuracy (after defining the operating range) of the detector are just two of the parameters that can be tested at a modular level. This will use a traceably calibrated digital flow meter and traceable wavelength standards. These are some of the modular tests that can be applied to the components of a liquid chromatograph but there should also be a holistic or system test that checks that the overall system works correctly as outlined by Furman et al.⁹ The holistic test is designed to show that the overall system works as individual modules may work as required, but errors can be additive and the system may not function as required.

The explicit statement that destroys the argument for SSTs replacing AIQ is also found in the same section of USP <1058>: Routine analytical tests do not constitute OQ testing.²

Let us examine and interpret this statement carefully. Routine analytical tests are defined in the top two layers of the data quality triangle shown in Figure 1: System suitability test injections and quality control check samples. These are based on a specific analytical method and do not test any explicit instrument parameter. Specifically, as shown by the following quotation: OQ tests are specifically designed to verify the instrument's operation according to specifications in the user's environment.²

This is reiterated by the USP's <621> on chromatography: To ascertain the effectiveness of the final operating system, it should be subjected to a suitability test prior to use. The essence of such a test is the concept that the electronics, the equipment, the specimens and the analytical operations constitute a single analytical system, which is amenable to an overall test of system function.¹¹

Note the phrasing used here — an SST is an overall test of system function. This is a point of use check that is run on the day rather than showing that the instrument is fit for its full intended purpose. This relies on the chromatograph having been qualified in the first place. However, you must also be aware that in HPLC many examples have been found of SST passing its criteria while in fact the instrument is out of specification for wavelength accuracy, flow accuracy, injection volume accuracy or temperature control accuracy.¹¹

As an SST cannot check these fundamental HPLC characteristics it fails totally as a substitute for a proper AIQ test and report which does qualify or calibrate these parameters. So to claim that an SST is AIQ is a circular and futile argument: an SST is method based and can only check if a specific method is working. If in doubt, look at the tests identified in the example warning letters presented earlier in this paper.

Conclusions

This column looked at the role of analytical instrument qualification to demonstrate that the instrument is fit for purpose across the operating ranges determined by the laboratory. This must be done before any methods are established or samples are assayed. Failure to qualify your instrument effectively (using test equipment that is calibrated to national or international standards) means that the remainder of the data quality triangle is meaningless and the results generated are questionable at best.

In contrast, a system suitability test is a point of use check to confirm that the instrument and the analytical method are working correctly just before the analysis begins. As such, an SST is totally dependent on the instrument having been qualified. An SST cannot be a substitute for AIQ because it is method and not instrument based. Failure to follow this advice and just rely on system suitability tests to qualify your instrument not only leaves you at risk of regulatory action, but also significantly reduces the trustworthiness of your analytical results.

"Questions of Quality" editor Bob McDowall is Principal at McDowall Consulting, Bromley, Kent, UK. He is also a member of LCGC Europe's Editorial Advisory Board. Direct correspondence about this column to "Questions of Quality", LCGC Europe, Bridgegate Pavilion, Chester Business Park, Wrexham Road, Chester CH4 9QH, UK, or e-mail the editor, Alasdair Matheson, at amatheson@advanstar.com

References

1. R.D. McDowall, LCGC Europe,23(7), 369–374 (2010).

2. United States Pharmacopoeia general chapter <1058> Analytical Instrument Qualification.

3. R.D. McDowall, LCGC Europe, 22(4), 204–210 (2009).

4. H. Avallone, Laboratory Controls and Compliance, in Analytical Chemistry on a GMP Environment, Eds J.B. Miller and J.B. Crowther, ISBN 0-471-31431-5 Crowther & Miller section 2.3.5 page 41 (2000).

5. Chemsource, FDA Warning Letter, November 2002 (www.fda.gov).

6. Earlham College, FDA Warning Letter, July 2002 (www.fda.gov).

7. Ion Laboratories, FDA Warning Letter, July 2009 (www.fda.gov).

8. ICH Q2 R1 Validation of Analytical Methods: Text and Methodology, 2005 (www.ich.org).

9. W. Furman, R. Tetzlaff and T. Layloff, JOAC Int., 77, 1314–1317 (1994).

10. United States Pharmacopoeia general chapter <621> Chromatography.

11. P. Coombes, Laboratory Systems Validation Testing and Practice 2002. ISBN Number: 1930114486.