Evaluating Analytical Instrumentation Using the Golf Score Card Tool

The purchase of expensive capital instrumentation is critical to the success of any analytical laboratory. In the process of selecting instrumentation, careful deliberation of the scientific, economic, and organizational factors, in the context of a laboratory's specific charter, is required to obtain the maximum benefit for the life of the equipment. Consideration of the important factors, in the context of our evaluations of HPLC–MS-MS systems primarily intended for quantitative bioanalytical applications, is explained. The use of a simple Excel-based "Golf Score Card" tool has facilitated our assessments and instrument acquisition decisions by collating and weighting the relative importance of the many criteria that impact the final purchase decision. This approach can be adapted readily to the evaluation of any analytical instrumentation. Additionally, the overall process is examined, practically and strategically, from the perspective of those who are both end-users and stewards of vital mass spectrometry-based technologies within our industrial health care organization.

Scientists using analytical instrumentation are faced with the need to evaluate and select technologies they will use to perform their jobs. In most cases, there are a number of instrument choices that could be purchased to perform the required task. However, within a given application and context, there will almost always be one selection that will have advantages over the alternatives. There is essentially no formalized training and little information in the literature (1–4) to guide scientists in the acquisition of critical tools.

High performance liquid chromatogrpahy (HPLC)–mass spectrometry (MS)-MS using triple quadrupole mass spectrometry is well-established as the technology of choice for measuring ultratrace concentrations of small molecules (drugs, metabolites, and biomarkers) in biological samples. Due to the vast array of bioanalytical needs spanning all stages of drug discovery, development, and market support (5), such quantitative measurements have grown to represent the single largest segment of support provided by our health care MS laboratories. In providing such support over many years, we have accumulated considerable experience not only in HPLC–MS-MS-based applications but also in continually assessing and optimally integrating new MS-based tools to improve capabilities and capacity to meet project needs. In this article, we describe our general philosophy and process for mass spectrometer assessment and selection. These same principles have also been applied to the acquisition of other types of instruments (such as liquid chromatographs and autosamplers).

A primary objective for instrument acquisitions has been to increase the capacity for addressing the growing needs across our health care businesses. At the same time, we continually seek to improve the technical capabilities of our bioanalytical arsenal. To deliver against bioanalytical quantitation objectives alone, we currently have 14 triple quadrupole HPLC–MS-MS systems across three laboratories, spanning pharmaceutical discovery, development, and over-the-counter R&D and manufacturing. A decision to acquire a specific instrument should not be taken lightly, especially when the price tag is up to $400,000 for a total system (with chromatograph and autosampler). Through each of the many deliberations we have had, for these and other types of mass spectrometers (about 60 instrument acquisitions over two decades), we have worked toward continual improvement in our process of assessing and selecting instrumentation. In addition to making the best choice for the business need at hand, attention is focused toward efficient use of our (and vendors') time and assures that evaluations will result in predefined measures that can contribute to an objective selection.

For experienced and established laboratories such as ours, the instrument evaluation can transition through several stages. Through vendor contacts and discussions with our counterparts at other companies, we first determine if various vendor offerings have evolved (improved) sufficiently since our prior instrument evaluation exercise, to warrant a multivendor instrument evaluation. If not, we move directly into value negotiations with the preferred vendor. If an evaluation is needed, contending instruments are selected, and a Golf Score Card tool is constructed. First-tier technical assessment often is conducted remotely (without travel) by submitting prescribed experiments to the vendor that will provide a clear assessment of their relative ability to meet our needs. If there is a clear winner that has hardware and software platforms that we are also very familiar with, we proceed without the cost and time of a demo trip. If the first-tier exercise uncovers a less-familiar leader (or no leader), we would only then opt to do a hands-on evaluation, which would include a more in-depth (comprehensive) set of experiments. For less established laboratories, an in-person evaluation of instrumentation would be recommended as the first course of action.

Multiple vendors are now able to market quality HPLC–MS-MS instruments (and other types of analytical tools), each offering somewhat unique attributes in order to differentiate their product (or find a special niche) relative to the others. Obviously, this benefits endusers in several ways. The resulting competition tends to drive list prices down over time and strengthens customer leverage in value or pricing negotiations at the point of purchase. This situation also has fueled vendor R&D efforts, further accelerating the evolution rate in instrument enhancements, providing access to increasingly effective HPLC–MS-MS tools for bioanalytical applications. That said, this situation does create a more difficult task in sorting through the pros and cons in the instrument selection process. We would argue that even though a number of different instruments could accomplish a given analytical task, if one considers the cost of the capital investment and the length of time an organization will utilize (and depreciate) that selection, even a small overall advantage that one system demonstrates relative to others will provide a significant benefit over the life of the analytical tool (typically 7–10 years for mass spectrometers and chromatographs).

Recognizing this dilemma, we evolved toward a systematic process for objectively measuring the relative value and appropriateness of candidate instrumentation for the intended use. The process involves four steps:

• Develop a list of primary factors that might influence the relative effectiveness of an instrument in its intended application area. This includes not only the usual analytical attributes (sensitivity, linear range, and so forth) but also factors such as software versatility, quality and affordability of vendor service, receptiveness of vendor to meet our specific needs, and especially practical implementation considerations such as our familiarity with hardware and software platforms.

• Stratify these factors into one of three categories, prioritized in terms of relevance (importance) within the context of the intended application area (most important, important, less important).

• Collect data that allows a rank-ordering of the contending instruments in terms of each factor. Depending upon the factor, data might be collected as part of the experimental evaluations or through a survey of the experiences of others (for example, quality of vendor service).

• Tabulate the results and weight each value, according to the importance category within which the factor resides.

We then total these weighted values for each instrument. Low score wins. Thus, we have termed this the "Golf Score Card" tool. This is admittedly not an exact science, however, we have found it to be a practical tool not only to help us sort through myriad considerations, but also to distill findings more clearly into a form that can be digested readily by our managers, researchers, and technical peers throughout our company. We explain here in detail our process and rationale. We show results from an extensive evaluation but do not name specific manufacturers because we intend to describe our process and not a relative evaluation that is in a sense a snapshot at a point in time. Instruments are continually evolving and a specific evaluation will be less relevant the following year.

Selection Criteria

To use the Golf Score Card process, regardless of technology being acquired, careful consideration of the key criteria is required. For our first example, triple quadrupole MS-MS systems for quantitation, 10 technical or vendor factors were identified as being important in the instrument selection.

Sensitivity: The smaller the amount of target compound that can be detected and quantified from a complex biological matrix, the more problems can be solved quickly and with less overall effort. For example, more sensitivity often translates to less sample preparation or allows additional dilution (reduction of matrix). Certainly, other factors contribute to the bottom-line sensitivity (for example, selectivity, ion source ruggedness), but in our view, this represents the composite of the single most important HPLC–MS-MS performance attribute for quantitation.

Common platform: When a primary objective is to maintain (through replacement) or increase capacity within a given application area, significant efficiency advantages are realized by maintaining common (identical) hardware and software platforms. A few positives gleaned in the areas of training multiple users are method troubleshooting and transfer (for example, from discovery to development); instrument repair (part swapping) and spares stocking; and ease of maintaining regulatory compliance. Moreover, we have a "knock out the champ" philosophy. All other things being equal, we would go with the technology we already know and can utilize seamlessly and quickly. This is one example of why context is such an important qualifier in this process. In the case of a lab purchasing its first HPLC–MS-MS system, this factor would have no relevance at all, but to us it is a very important consideration.

Software–Data System: Ideally, this is user-friendly and intuitive. Power and versatility of software must allow streamlining of data acquisition, reduction, reporting, and archiving. Also, the ability to set-up and control parameters of peripheral tools such as chromatographs and autosamplers is critical to exploiting increasingly automated sample and data management capabilities.

Linear dynamic range: The ability to analyze samples over a wide range is key to supporting pharmacokinetic studies in which target compound levels can vary by over 1000-fold across study samples.

Ion source–interface: The volatilization–ionization process must cause minimal analyte degradation and be applicable across a broad range of compound classes. Ion sources that can handle a wide range of liquid flows without compromising performance provide versatility for high-throughput applications and are readily adaptable to a broad range of chromatographic inlet technologies are obviously advantageous. Other factors include probe or mode switching (for example, from electrospray [ESI] to atmospheric pressure chemical ionization [APCI]).

Up-time–ruggedness: Minimal maintenance and downtime are ideal for optimal instrument performance and throughput. This can be ascertained from experience and discussions with other instrument users (internal and external).

Vendor service: This often is determined from the recent track record of field engineer responsiveness and experience. There usually will be local variability in terms of quality and responsiveness, as service usually is directly related to the ability of a few individuals, but vendor organization and cost are also factors.

Cost: This is a factor that definitely plays into a purchase decision, although to what extent will vary by organization. However, one way to look at this is to consider it in terms of value. Lower cost for equivalent performance is better, but cost ideally is not the primary driver. Paying $100,000 less for an instrument that cannot meet the business needs optimally is not a cost savings. It is a waste of money.

Influence on Vendor: This reflects the vendor's track record of working with us to address technical needs specific to our applications (for example, provide custom programming to control a preferred accessory) or in defining next-generation software and hardware improvements.

Mass Resolution–Accuracy: This consideration was added because of the potential of this advantage (6). Improved mass resolution capabilities (better than nominal) provide a potential means of increasing specificity for quantitative applications. This capability could be advantageous when detection limits are chemical-noise limited and the target compound contains one or more hetero atoms (such as P, Cl, and Br) so that the analyte has a negative mass defect relative to the predominating biological matrix components.

The preceding are the factors we value for consideration of mass spectrometers intended for quantitation of potential pharmaceutical and related compounds. For some types of instrument acquisitions, we also deliberate on potential instrument redeployment and life-cycle management considerations. Other organizations might consider additional factors including GLP compliance, interfacing to LIMS, and manufacturer stability–viability. Evaluation of different types of instrumentation necessarily will involve other considerations. For example, for an evaluation of autosamplers for ultratrace quantitation, injection-to-injection carryover would be an important consideration. For HPLC systems performing that job in conjunction with MS-MS, extracolumn volume, retention time reproducibility and data system interfacing would be considered. An important point to reiterate is that, for each type of instrumentation being considered, it is absolutely essential to first clearly define the context or task for which that instrument is targeted to devise the optimal list of evaluation considerations and weight them appropriately.

The Golf Score Card

Information that allowed a rank-ordering of contending instruments, in terms of each factor, was collected in various ways. Four factors (sensitivity, ion source–interface, linearity–dynamic range, and mass resolution) were addressed by data obtained from vendor (and in-house) execution of our prescribed experiments (see next section). Five more subjective factors (vendor service, common platform, up-time–ruggedness, influence on vendor, and software–data system) were assessed through surveys and discussions with users (internal and external) having the most relevant experience. The cost factor reflected the standard corporate discount from each vendor (first-cut negotiation). To yield the "Golf Score Card," all rank orderings were entered, weighted, and then compiled using an Excel spreadsheet.

The Golf Score Card, shown in Figure 1, was constructed from an evaluation conducted a few years ago. The totals of all ten of these weighted factors provided a measure of relative fit of each instrument toward meeting the lab's requirements. The rankings that were considered most important were multiplied by three, so as to give these the most influence on the final score. Those considered important were multiplied by a factor of two. The golf score (weighted summation) for each instrument was calculated using a simple Excel formula (shown here for Instrument 1):

=SUM(B3:B4)*3+SUM(B6:B10)*2+SUM(B12:B14)

Figure 1: The Golf Score Card showing data from an evaluation of three triple quadrupole MS systems. The low "golf" score won and Instrument 1 was purchased. Weighting: most important category multiplied by 3 and important by 2, with data summed by the spreadsheet.

As with the game of golf, the instrument with the low score wins. The score or weighted summation was influenced by each category, but to different extents based upon level of importance (associated weighting). In some cases, a straightforward ranking of first, second, and third place was not used. For example, instruments 1 and 3 were found to have equivalent sensitivity and instrument 2 was significantly less sensitive. Therefore, 1 and 3 received a 1.5 ranking (splitting the difference between first and second place) and instrument 2 a third place ranking. In a similar manner, as we already had several (previous generation) instruments from vendor 1 and no triple quadrupoles from 2 and 3, we assigned a first place ranking to instrument 1 and 2.5 rankings to the other two. Clearly, it would be simple to configure a Golf Score Card to reflect individual needs and emphasis. Categories and weightings can be adjusted easily depending upon instrumentation type, researcher experience, existing equipment and, importantly, the specific charter or task at hand. That we configured and weighted our Golf Score Card the way we did is a reflection of our context. This context included our experience, instrumentation on hand, lab charter, and so on. The cost differential was low (in this case) and up-time–ruggedness was based upon previous generation instruments, so these factors were de-emphasized by placing them in the less important category. The primary point is that this simple Excel-based tool allows multifaceted data to be well-organized and encourages consideration of the relative importance of the various factors that lead to a decision.

Additionally, we conducted a recent and more extensive evaluation of MS-MS quantitation systems that included ion trap mass spectrometers and additional evaluation categories (mass accuracy, full-scan sensitivity, high-flow source capability). A retrospective assessment of this evaluation data revealed a direct correlation between our golf scores and instrument pricing. The linear plot of the scores versus cost (from initial price negotiations) shown in Figure 2 supports this. The fit quality (r = 0.9108) from our golf score data implies that vendors already know where their instruments rank against each other in terms of bioanalytical quantitation capability (a large application area for multistage MS systems). Furthermore, the resulting linear fit might be thought of as a "Value Line." Scores below the line tend to be good values, while those above the line are perhaps overpriced for what they might deliver in health care-oriented quantitative MS applications. Note that the golf scores from the more recent evaluation (Figure 2) are higher than those from the first example (Figure 1). This was simply a result of the math from additional categories (three more) and more instruments in the evaluation (that is, fourth through eighth place values also possible).

Figure 2: This golf score versus price correlation shows the relationship between cost and the relative bioanalytical quantitation capabilities of eight mass spectrometers.

Conducting an Instrument Evaluation

It is important to construct evaluation or "demo" experiments that directly assess technical performance in the application areas the instrument will be used for. In order to reduce variables, we are prescriptive in detailing experiments and in the formatting of results that the vendor must provide for our review. Collectively, this maximizes the chance that observed differences in instrument performance are not a reflection of location, demo chemists, or other factors we do not wish to test. A significant part of this is avoiding the "apples and oranges" dilemma that arises from ancillary variables that a vendor might impart as they execute the experiments, or the manner in which individual vendors might choose to display the data. For example, to eliminate chromatography variables in our assessment of mass spectrometers, we employ isocratic HPLC methods for analyte introduction and even provide the vendor with HPLC columns (all purchased from the same lot). We also mandate that vendors precisely adhere to our prescribed experimental parameters and procedures and provide a page-by-page listing of the key figures (spectra and chromatograms) that they must assemble and present to us at the conclusion of their experiments. Another way we minimize difficulties in comparing performance attributes is to select test compounds with which we have considerable historical experience. This not only gives us confidence that these experiments will work if executed as prescribed (very low uncertainty), it also permits us to generate the best possible in-house "control" data in our hands (on our existing instrumentation).

Isocratic HPLC–MS-MS assays for dextromethorphan and fluprostenol were chosen as the basis for some of our recent instrument evaluation experiments (Figure 3), including the two examples discussed in Figures 1 and 2. Over the years, we have analyzed tens of thousands of biological samples containing one or the other of these compounds. Dextromethorphan, an ingredient in over-the-counter cough and cold products, represents a typical drug active. Its tertiary amine substructure is protonated easily via ESI and yields abundant and structurally specific fragment ions under typical MS-MS conditions, making it a compound that can typically be detected with high sensitivity, across a variety of MS-MS instrument types (7–10).We have found that dextromethorphan is an analyte that any system can determine, yet every system it has been tested on responds differently, allowing an interesting and reliable comparison. Fluprostenol represents a more difficult class of compounds to analyze by positive ion ESI-MS-MS detection (11,12). Fluprostenol ions are much more labile than dextromethorphan. [M + NH₄]⁺ ions formed under all but the mildest of ESI conditions tend to degrade within many ESI source/interface types ([M + H]⁺ ions are not abundant). This makes fluprostenol an ideal compound to test ion source–interface quality and provide an indication of the breadth of compound class applicability of a given instrument.

Figure 3: The structures of two compounds used for MS-MS instrument evaluations, dextromethorphan and fluprostenol.

Conducting an instrument evaluation at the vendor's site is an excellent opportunity to learn about their instrumentation and software. Demonstration lab chemists often are highly experienced and have a varied perspective on the type of work different customers intend to perform with their new instrumentation. Often, the prospective customer will have access to vendor experts and learn about the company and its support structure, so that overall these visits can be an important learning experience. After a number of iterations of the demonstration and acquisition experience, with the same vendors and instrument types, an on-site demo might not be required. We have managed the acquisition of 15 triple quadrupole HPLC–MS-MS systems since 1992. Although improved systems continue to be offered, this technology is fairly mature and we have a great deal of experience with it. In the absence of a revolutionary (as opposed to evolutionary) improvement or a significant offering by a vendor that we are less familiar with, our last several rounds of acquisitions have been conducted with assays sent to demo labs with the aforementioned carefully prescribed experimental conditions. In cases in which no significant evolutionary change has occurred and the acquisition might be tied to capacity needs, a paper argument will suffice (no demo at all).

It is important to consider the many other sources of information that are readily available, in addition to a direct demonstration of instrumentation. These include vendor websites, literature, and, more recently, net meetings or "web casts" (conferences or presentations by vendors to groups over the internet). Other important sources of information include trade shows (PittCon, for example) and informal discussions with colleagues, counterparts from other companies, and contract lab personnel.

Conclusions

The acquisition of expensive and important analytical instrumentation always is preceded by a significant decision. Typically, there will be several instrument choices that can perform the required analyses to different degrees. In addition, we would argue that the purchase decision impacts business success, careers, and future acquisition opportunities and therefore, even a small advantage, especially over time, is of concern to the successful scientist. Careful consideration, categorization, and weighting of the relevant factors can greatly increase the chance of an ideal outcome. Our experience is that the use of a tool such as the Golf Score Card helps focus, organize, and streamline this process. We cannot emphasize strongly enough the importance of carefully considering potential acquisitions within the context of the intended application. Instruments can have scientifically interesting attributes that do not directly impact the work an instrument is intended for and these factors should be minimized in an evaluation. Importantly, the demonstration and evaluation process is a key learning opportunity. Insights into instrument usage or related techniques can be applied in future efforts.

Acknowledgments

The authors would like to thank Tom Eichhold, Mike Quijano, Mary Kay Dirr, Pete Stoffolano, Dave Foltz, Michelle Dunaway, John Tomlinson, Adam Lux, and Rose Marie Deibel for assistance during our recent instrument evaluations and acquisitions.

Tim Baker has been with Procter & Gamble since 1990. After five years in the Health & Personal Care and Over-The-Counter Technology Divisions, he joined P&G Pharmaceuticals. He is currently principal scientist and group leader of the Bioanalytical MS Laboratory, and his research interests include finding new ways of supporting pharmaceutical discovery with mass spectrometry-based technologies. Tim received the Ph.D. in analytical chemistry from Northeastern University, Boston, Massachusetts, in 1990.

Steve Hoke joined Procter & Gamble Pharmaceuticals in 1994. After one year in Norwich, New York, he moved to Cincinnati, Ohio and joined the Oral and Personal Health Care (O&PHC) organization of Procter & Gamble. He is currently a principal scientist and leader of the O&PHC mass spectrometry group.Steve received the Ph.D. in analytical chemistry from Purdue University in 1994, where his graduate research focused on applications of mass spectrometry.

Roy Dobson joined Procter & Gamble's Health & Personal Care Technology Division in 1986. He is currently a research fellow and technical manager of Health Care Mass Spectrometry. Roy's research interests touch on most aspects of MS-related innovations, with greatest emphasis in the areas of ultratrace quantitation, drug metabolism, and high-throughput small molecule characterization. Roy received the Ph.D. in analytical chemistry from Iowa State University, Ames, Iowa, in 1986.

*Procter & Gamble Pharmaceuticals, The Procter & Gamble Company

†Health Care Research Center, Mason, Ohio.

Please direct correspondence to Tim Baker at baker.tr@pg.com

References

(1) J.R. Jordan, Managing Modern Lab. 1, 29–34 (1995).

(2) J.S. Kroger, Consultant 27, 71–72, 75 (1987).

(3) E.F. McFarren and R.J. Lishka, J. Water Pollution Contr. Fed. 43, 67–72 (1971).

(4) A. Coddington, J. Van Antwerp, and H. Ramjit, J. Liquid Chromatogr. Rel. Technol. 26, 2839–2859 (2003).

(5) S.H. Hoke, K.L. Morand, K.D. Greis, T.R. Baker, K.L. Harbol, and R.L.M. Dobson, Int. J. Mass Spectrom. 212, 135–196 (2001).

(6) X. Xu, J. Veals, and W.A. Korfmacher, Rapid Commun. Mass Spectrom. RCM 17, 832–837 (2003).

(7) R.D. Bolden, S.H. Hoke II, T.H. Eichhold, D.L. McCauley-Myers, and K.R. Wehmeyer, J. Chromatogr., B 772, 1–10 (2002).

(8) T.H. Eichhold, L.J. Greenfield, S.H. Hoke, and K.R. Wehmeyer, J. Mass Spectrom. 32, 1205–1211 (1997).

(9) D.L. McCauley-Myers, T.H. Eichhold, R.E. Bailey, D.J. Dobrozsi, K.J. Best, J.W. Hayes, and S.H. Hoke, J. Pharm. Biomed. Anal. 23, 825–835 (2000).

(10) T.H. Eichhold, M. Quijano, W.L. Seibel, C.A. Cruze, R.L. Dobson, and K.R. Wehmeyer, J. Chromatogr., B 698, 147–154 (1997).

(11) T.H. Eichhold, D.L. Kuhlenbeck, T.R. Baker, M.E. Stella, J.S. Amburgey, M.A. deLong, J.R. Hartke, C.A. Cruze, S.A. Pierce, and K.R. Wehmeyer, J. Chromatogr., B 741, 213–220 (2000).

(12) R.D. Gauw, P.J. Stoffolano, D.L. Kuhlenbeck, V.S. Patel, S.M. Garver, T.R. Baker, and K.R. Wehmeyer, J. Chromatogr., B 744, 283–291 (2000).