Implementing a Mission Critical Client/Server Chromatography Data System for Continuous Uptime Application of an Enterprise-Level CDS and Microsoft Cluster Service

For many chemical and pharmaceutical companies, rapid and continuous dissemination of laboratory information can have significant business impact on product and process quality. Throughout the production cycle, analytical data characterizes raw materials, intermediates, and finally the product itself. Chromatography remains a key technique for determining product quality and for identifying when products are trending towards nonconformance. At the heart of this analysis is the chromatography data system (CDS). Whether the sample analysis is performed by gas chromatography (GC) or liquid chromatography (LC) instrumentation — and whether the analysis is conducted by quality assurance, research, or process operations, the CDS software that controls and processes these chromatographs provides information that drives critical business decisions.

In manufacturing critical applications, these instruments and their associated CDS software must provide the analytical information in a continuous workflow. Chromatographic analysis might be requested at any time and from different locations in a manufacturing operation. A laboratory that is unable to provide CDS results on demand can have a direct impact on the overall efficiency of that manufacturing operation and can impact a company's financial performance adversely. To meet these demands for continuous operational capacity, laboratories are staffed as efficiently as possible. As an illustration of this, it is not uncommon for a single operator to be responsible for 10 chromatographs or more. Therefore it is critical, when implementing a CDS, to take into consideration both the needs of the business and the capacity of the laboratory.

Integrating the chromatography laboratory in a manufacturing-critical application involves careful planning in a number of key areas. Maintaining proper instrument availability and operation, and utilizing a robust and reliable computer architecture and network are major considerations. In addition, the availability of the CDS hardware (network controllers, local area network [LAN] devices, computer servers, client computers, and so forth) can be a major consideration. Both "unplanned" and "planned" downtimes of the CDS hardware occur. Unplanned downtimes can be due to unforeseen circumstances, such as a hardware or software failure, or failure of an external agent, such as a network. Planned downtimes involve scheduled maintenance, installation of operating system revisions and patches, and similar routine activities. In implementing a CDS in a manufacturing-critical environment, managing these planned and unplanned downtimes for peak operational efficiencies must be a critical consideration.

Finally, any solution providing CDS uptime and reliability ideally will have minimal operational impact on end-users. Any change in the CDS operation to improve uptime and reliability should be "transparent" to end-users so that the way they collect data, control instruments, and report results is based upon typical and not artificially imposed workflows.

By pairing the right CDS software and instrument controllers with the right solution, laboratories can meet the needs of their manufacturing operations successfully to maintain continuous uptime. This case study describes the successful deployment of a scalable CDS in a manufacturing-critical environment that delivers continuous CDS operation during unplanned and planned downtimes.

Description of the Business Environment

The company illustrated in this business case is one of the world's largest manufacturers and one of the world's largest integrated producers of polymers and fibers operating in more than 20 countries across North America, South America, Europe, and the Asia-Pacific region. The company selected an enterprise CDS (Atlas CDS, Thermo Fisher Scientific, Philadelphia, Pennsylvania) to control and process data from the more than 60 gas and liquid chromatographs in their process control laboratory. The laboratory's main analysis is coupled closely to testing and monitoring plant manufacturing operations. In this workflow, the continuous availability and operation of the instruments and their data processing through the CDS are part of a manufacturing-critical operation. The instruments are intended to operate in a continuous environment among all the laboratory's operators. A typical chromatographic analysis would be expected to occur in 30 min or less during routine work operations.

The CDS is used to provide distributed instrument access, chromatography data acquisition from the gas and liquid chromatographs, and data calculations that are needed for management to assess the quality and performance of various manufacturing processes. The company also utilizes a laboratory information management system (LIMS) to notify the laboratory of work requests and the instrument analysis required for each sample, relying on the built-in integration between the CDS and the LIMS to automate this process. Once the analysis is complete, the CDS results are then incorporated with other analytical results and checked against a target specification in the LIMS. These results, along with additional information, are then delivered to the enterprise resource planning (ERP) and process control systems to facilitate informed management decisions and process control changes.

The Challenge

As part of a laboratory modernization initiative of hardware upgrades, this company was in a position to consider further improvement in terms of making their CDS deployment more robust and reliable. In modernizing their CDS data acquisition architecture, instrument controllers for the CDS were upgraded to the latest controller variant, the "24/7." This controller variant provides automated instrument control and data acquisition for up to four instruments on a LAN simultaneously. Under normal circumstances, the instrument controllers control each chromatograph, and data is collected in real time. Following run completion, the run is processed, analyzed, and reported. The 24/7 instrument controllers are able to engage the connected LC and GC systems independent of the CDS server so that data acquisition is continuous, even in the event of a communication failure with the CDS server due to network or hardware outage. The 24/7 instrument controllers will run any samples that have been submitted in the order they were requested, thereby preventing data loss. Once the connection to the server is restored, the 24/7 instrument controllers automatically send the data to the CDS server for processing and submit final results to the LIMS.

Through implementation of the CDS and the 24/7 instrument controllers, the company had built a robust laboratory implementation workflow. The next step was to design and implement a solution that would provide redundancy without creating an unnecessary burden on the information technology (IT) department. To help them with this solution, the company turned to Computer Sciences Corporation (CSC), Falls Church, Virginia.

Planning for Uptime and Reliability with Microsoft Cluster Service

CSC, a global solutions provider, is currently working with several of the company's sites for LIMS support. The company engaged CSC to assist in providing a solution that would reduce the incidence of single point of failure for their existing CDS system. CSC helped provide a more reliable CDS workflow and suggested improvements that would ensure the reliability and availability of the CDS server. While the laboratories had a back-up server in place, switching to it was a manual process. In initiating this effort with CSC, the company outlined two key requirements: the solution must be automated so that manual intervention was not required to switch between the live server and the backup server; and, the solution must be transparent to the company's CDS users and have no impact on the operations of the laboratory or on the plant relying on the data.

Clustering is used frequently in scenarios where a transparent transfer of service is required from one piece of hardware to another. It, thus, acts as an ideal solution for environments such as those found in a global manufacturing setting, addressing the problems of both planned and unplanned downtime. The technical name for this transfer of service is "failover." When using clustering to support server failover, the cluster is typically a group of servers that appear to the outside world as a single server. While processing of data takes place on individual servers within that cluster, the aim of clustering is that users of the cluster should neither know nor care which actual server in the cluster is doing their work for them.

In this situation, the cluster consists of two physical servers, node A and node B. These servers are Quad Core Xeon Processors running the Microsoft Cluster Service (MSCS) to host the Client/Server CDS application. Using the Microsoft Cluster Service, these two nodes essentially appear to the outside world as a single node, and the CDS Alias is the means to refer to this single, virtual server. At any given time, only one of the two nodes is actually active and processing CDS data. The second node is there as a backup.

If for some reason the active node fails, or services on it fail, the Microsoft Cluster Service can invoke the same services transparently to start operating on the second node, and all communications that were previously going from the CDS Alias to the first node are now directed to the second node. Thus, for all intents and purposes, there appears to be no break in services from the outside world, because all external agents are connected to the CDS Alias rather than to node A or node B directly. This entire transfer of execution of CDS services from node to node is entirely automatic and invisible to the users and instruments availing of the services.

As part of the CSC solution, a notification process was embedded in the clustered application that automatically sends a notice to key laboratory and IT personnel by e-mail, cell phone, or pager informing them that an unplanned failover has taken place. In any laboratory setting, unplanned failovers are rare, but they do occur. This notification mechanism ensures that laboratory managers can take corrective action and address the problem so that it does not occur in the future.

For planned failovers, the laboratory schedules their critical patches and system updates once a month. Before implementing their cluster environment, failover required manual intervention and interrupted laboratory processes. With the implementation of the CDS on Microsoft Cluster Service, this failover can be handled by the IT department without impacting the laboratory at all and does not require any intervention from the CDS Manager. The only time the CDS Manager does need to perform failover is to take advantage of the latest enhancements to the CDS itself. To perform a planned failover, the IT department or CDS Manager simply log into the node that owns the CDS application and open Microsoft Cluster Administrator. The Cluster Administrator lists all of the cluster groups defined on the server, and it is a simple matter to select the CDS Group and perform the failover.

Summary

With the use of MSCS, the laboratory's CDS users are completely unaware that a failover is occurring and there is no interruption in data acquisition. The CDS enables users to process samples in real-time or at the completion of the run. For most of the work done in this company's laboratories, the data is auto-processed at the end of the run, but for critical samples, real-time processing is essential. The use of the CDS on MSCS does not impact this real time processing, so mission-critical results can be obtained and the plant can be notified if process changes are required.

Without Microsoft Cluster Service, the company's deployment would require more dedicated system management and some manual actions to redeploy servers, redirect and reboot the instrument controllers, and it would involve human IT intervention. By using the combination of the redundant servers and MSCS, unplanned failovers occur automatically without intervention and both planned and unplanned failovers are invisible to the end-users running and engaging the CDS. The solution created by CSC delivers a very high level of confidence in the availability and operation of the chromatography data system during planned and unplanned failovers.

Based upon the implementation put in place by the company, the CDS operates with complete uptime, even when a server system is down due to scheduled or unscheduled system maintenance events. Most importantly, the CDS users are completely shielded from system operations such as an operating system maintenance patch. Users can continue to operate the CDS to create methods and sequences, address available instruments for data acquisition, and produce CDS data analysis and reporting. Retrieval of stored data and previous instrument methods can occur even if the original CPU containing the original data is unavailable due to the continuous shadowing of the MSCS environment.

Laboratory managers at this company now believe their experience and successful deployment of their CDS and Microsoft Cluster Service for manufacturing-critical operations can serve as an example to other sites within their organization as well as other companies in similar circumstances. With careful planning for the user workflows, identification of critical needs, and successful applications of available industry tools, a remarkable application for a CDS with continuous uptime is available to any laboratory.

Trish Meek and Ed Long

Thermo Fisher Scientific

Please direct correspondence to trish.meek@thermofisher.com