Troubleshooting 101: Some Essential Principles of Effective Troubleshooting

In an upcoming installment of “LC Troubleshooting,” I will focus—for the first time in this column—on troubleshooting topics specific to two-dimensional liquid chromatography (2D-LC) systems. These systems are inherently more complex than systems for conventional (one-dimensional) LC, with more components and ways for problems to arise. Given this complexity, troubleshooting problems in 2D-LC systems can be more challenging. As I’ve thought about how to approach this, it occurred to me that a review of some of the essential principles of troubleshooting would be instructive. At a minimum, a systematic and disciplined approach to troubleshooting challenging problems makes them easier to solve. This notion is not exclusive to troubleshooting problems with chromatography systems. Early in my career, an influential mentor frequently commented on the value of the training he received in the United States Army as an electronics technician when he applied that to his work as a field service engineer specializing in mass spectrometry systems. A casual look at training manuals for the Army reveals how much emphasis is placed on the value of a systematic approach for increasing both the effectiveness and efficiency of troubleshooting efforts (1).

One of the books I possess on the topic of troubleshooting is Troubleshooting LC Systems, by John Dolan and Lloyd Snyder (2). Given that it was published in 1989, a lot of the details related to how pumps, detectors, and injectors work are obsolete because LC technology has evolved so much over the past three decades. However, the content of chapter four of their book, “Principles of Troubleshooting,” is timeless, in my opinion. I encourage LC practitioners who are looking to expand their knowledge of LC and hone their troubleshooting skills to pick up a copy of this book, or a similar one. What follows in this installment is a review of some of the key concepts of a systematic approach to troubleshooting, which are common to the book by Dolan and Snyder and other similar books on the topic.

#1: One Thing at a Time

One of the most important principles in any systematic approach to troubleshooting is that in the problem-solving process, we should only change one thing at a time, observe the effect of that change, and then decide what to do next in response to the observed effects. This is an approach that is fundamentally different from changing several things simultaneously and hoping that at least one of the changes is beneficial (sometimes referred to as the shotgun approach). The problem with the latter approach is that if we change several things at once there is no way to know which of the changes is responsible for correcting the problem. The shotgun approach to troubleshooting is expensive because it leads to the replacement of parts that are perfectly fine and not contributing to the problem. In addition, the shotgun approach does not lead to an understanding of the root cause of the problem. If the root cause is not understood, then we lose the opportunity to take steps to address the root cause, and potentially prevent future problems before they occur. An example is helpful to illustrate these points.

Example: Unexpectedly High Pressure

Measured at the Pump

Seeing unexpectedly high pressure measured at the pump is a common problem in LC, encountered by inexperienced and experienced practitioners alike. A higher than normal pressure—provided the correct mobile phase, flow rate, and column are being used—is usually a result of some abnormal obstruction of the flow path, such as a partially blocked capillary or inline filter. Once the user is convinced that the observed pressure is higher than normal, the first question is: Which component of the system is the culprit? In a typical system, there might be five to eight different capillaries in the flow path between the pump outlet (where the pressure is measured) and the detector outlet, and one or more inline filters. In the shotgun approach, we would change all of the connection capillaries and inline filters at once, replacing each part with a new one. Depending on the type of capillaries in use, this could easily cost $500–$1000. Doing so may resolve the obstruction and return the pressure to a more typical value. However, at this point, we would not know which of the capillaries or filters was obstructed, thus missing valuable information.

If we instead take the one-at-a-time approach and check the pressure after each capillary is removed one at a time, starting from the downstream (detector) side of the flow path, we can clearly identify which capillary or filter is obstructed. This process not only localizes the repair or replacement that is needed, but may also yield clues about the root cause of the obstruction. For example, if we find that the capillary connected directly to the pump outlet is obstructed, it may be due to the pump seals shedding particulate material that accumulates in the capillary, or a badly contaminated bottle of mobile phase. If the needle seat capillary on the downstream side of the autosampler needle is obstructed, it may be due to samples that have particulate matter in them and need to be filtered. Finally, if we find that the inline filter downstream from the autosampler is obstructed, it may be due to shedding of seal material from a valve in the sampler.

The shotgun approach can be attractive because it is the most efficient approach to resolving a problem. Changing one part or variable at a time can consume an inordinate amount of time and analyst effort. In critical situations, we may not have the luxury of taking this time to really understand the problem. If the cost associated with changing many parts at once is not a concern, then doing so may be a viable approach even if we shortchange ourselves in the long run by not addressing the root cause of the problem that may lead to repeated problems of the same kind in the future.

#2: Do No Harm

Each laboratory has a slightly different approach to maintenance and repair of its instrumentation. On one end of this spectrum we have laboratories where every piece of equipment is covered by a service contract such that the laboratory staff are rarely asked to deal with performance problems. Rather, the instrument manufacturer or a third party specializing in such service is called upon to come in and address the problem quickly. On the other end of the spectrum, some laboratories don’t have any service contracts or warranties on their equipment, and do not have more than one of any particular piece of equipment. This is a particularly challenging environment for troubleshooting because spare parts are usually in short supply and the laboratory staff do not have a second instrument to use as a reference point when investigating unusual behavior of a particular instrument.

However, there is a middle ground worth discussing. Some laboratories have more than one of a particular make and model of an LC instrument. This is an important asset in the troubleshooting process because a functional instrument can serve as a reservoir of known working parts when troubleshooting another instrument of the same kind. However, discipline is important here in terms of how moving parts between instruments is handled. The principle is that parts that are “borrowed” temporarily from a working instrument for the purpose of troubleshooting another instrument should always be returned to the working instrument at the conclusion of the troubleshooting exercise. If this practice is not followed, confusion can result quickly, especially if the process is not documented carefully.

An example is instructive here. Suppose that the baseline signal of a UV detector suddenly becomes much noisier than is normally observed. There are multiple possible causes of this, including an aging deuterium lamp and dirty flow cell windows. There are multiple reasonable approaches to troubleshooting this problem, but if a similar instrument is nearby that has a working lamp with the same part number, then it is a simple and quick matter to move the good lamp from the working instrument to the instrument having the problem, and see if the noisy baseline is resolved. If the baseline improves, then this suggests that the lamp had reached the end of its life and should be replaced. At this point, we have a decision to make. Do we leave the lamp that we moved from the working instrument in place and install a new lamp in the instrument from which the used but working lamp was borrowed, or do we return the used lamp to the working instrument and install the new lamp in the instrument where the problem was originally observed? In this case, the best practice is to return the used lamp to the instrument from which it was borrowed. Being disciplined about this will minimize confusion that can occur in challenging troubleshooting exercises where multiple parts are moved between multiple instruments. Keeping parts with their “parent” instruments helps keep preventative maintenance schedules intact.

#3: Drawers Are Not Repair Centers

One behavior I’ve observed in many laboratories is that sometimes people are reluctant to throw away bad parts and components at the end of a troubleshooting exercise. For example, suppose that we have systematically worked to resolve a larger-than-normal pressure fluctuation observed with a pump. We first changed the inlet check valve, which did not solve the problem, so we put the old one back in. Then, we changed the outlet check valve, and the pressure fluctuation problem went away. We can conclude from this observation that the outlet check valve that was in the pump does not function properly. At this point, this failing part should be thrown away unless there is a clear path to actually fixing the part. In other words, simply putting the part in the drawer won’t fix it; rather, putting it in the drawer just creates a problem and questions for someone else to deal with later on. The function of check valves can occasionally be restored by sonicating them in isopropanol. However, what I sometimes see is that rather than addressing the problem with the part right away, it goes in a drawer, sometimes with a label that indicates it is a suspect or failing part, and sometimes not. Of course, not labeling the part as problematic is the worst-case scenario because an unsuspecting analyst may happen across the part at some later time. If she uses the part in a troubleshooting exercise, she will be doomed to fail from the start because the part is not functional. This problem can be avoided entirely by being disciplined about discarding parts that have been shown to be bad in the process of a thorough and systematic troubleshooting investigation. One exception to this is a case where the part or assembly is expensive and can be returned to the manufacturer for repair or refurbishment rather than simply discarding the part or assembly entirely.

#4: Build Up Knowledge

About Expected Behavior

A first step in solving any problem with an instrument is recognizing that there is a problem or behavior that does not look normal. Typically, establishing what normal behavior looks like is accomplished using two different types or sets of methods. At the time of installation of the instrument, a set of operational qualification (OQ) and performance verification (PV) methods are run that demonstrate that the instrument meets or exceeds the manufacturer’s specifications for performance parameters such as flow rate accuracy, gradient composition accuracy and precision, and detector drift, for example. At the time of installation, we may find that the flow rate accuracy under the test conditions is 99.7%. This method can be rerun at any point when a problem related to flow rate is expected, to see how the performance compares to when the instrument was installed. Readers interested in learning more about these types of methods and tests are referred to the July 2020 installment of “LC Troubleshooting” where Tony Taylor and I wrote about these in more detail (3).

A second type of method that can be used to look at instrument behavior over time is usually referred to as a system suitability test. Whereas the OQ and PV tests referred to above are set by the instrument manufacturer and don’t typically involve an actual separation (for example, the flow rate accuracy is evaluated without a column connected), a system suitability test is a method that involves conditions very close to the analysis at hand. Because this method will involve an actual separation, data on retention times (absolute times, as well as variability), peak shapes, and resolution values are typically recorded. Here again, this system suitability test can be run at any time (usually the frequency with which this is done is established as part of the method, or the laboratory culture) and the resulting values can be compared to previous system suitability test results to see if the current results look like they are out of line. Readers interested in learning more about this type of test are referred to previous installments of “LC Troubleshooting,” such as reference (4).

#5: Documentation

Helps in the Long Run

The details of any troubleshooting exercise should always be recorded in a laboratory notebook or logbook associated with the instrument. However, there are two aspects here that I think are noteworthy. First, tracking changes of commonly changed parts such as pump seals, valve rotor seals, and UV lamps can help assess the likelihood of failure in the future when one of these parts is suspected as being the cause of a problem. For example, if I change the seals in a pump today and then observe a leak from the pump two years from now, I would say that it is a strong possibility that the seals are the source of the leak given their age, and should be changed, but I would only be able to come to this conclusion if I had documented the prior change of the seals so that I can look in the record later on and see when the change occurred. In other words, documenting changes today will make troubleshooting new problems easier in the future. Second, there are many ways for things to go wrong with LC instruments and many times the observed problem appears to be truly unique. Given enough time and work experience with enough instruments, we can start to observe the same problem multiple times. Carefully documenting troubleshooting exercises over time facilitates accumulation of local knowledge about particular problems that are commonly observed with each instrument’s make and model as well as known solutions to those problems.

#6: Know When to Ask for Help

As discussed above, different laboratories can have very different cultures and expectations with respect to instrument maintenance and repair. In some cases analysts are asked not to attempt any repairs at all, and all of this work is handled by a third party contractor. In other cases analysts are expected to solve all but the most difficult problems on their own, with the resources in the laboratory. This approach has limits, of course, and increasingly modern instruments are becoming less and less approachable when it comes to troubleshooting. User and service manuals are not as detailed as they were in the past, and users are provided with less guidance from manufacturers about potential solutions that can be explored when a problem of a particular type is encountered. The circumstances surrounding many troubleshooting exercises are unique, and thus it is difficult to provide general advice about when to reach out for professional help. If all of the troubleshooting guidance available to you in the manual for your instrument has been exhausted and you still don’t have a solution to the problem, don’t hesitate to reach out for help from the manufacturer’s service organization. Their intimate knowledge of their systems can enable quick solutions to problems that appear insurmountable to us. At some point leveraging the expertise of the manufacturer—though it may be expensive to do so—is the best option to solve the problem quickly and get back to analyzing your samples.

Summary

In this installment of “LC Troubleshooting,” I have discussed some of the essential principles of a systematic approach to troubleshooting problems with LC instruments. On their own each of them is conceptually straightforward and easy to implement in practice. However, taken together, and with practice, incorporating these ideas into your troubleshooting approach will improve both the efficiency and effectiveness of your troubleshooting work.

References

(1) US Army Intelligence Center, Introduction to Logical Troubleshooting (1996). https://bit.ly/342c1fg (accessed September 7, 2020).

(2) J.W. Dolan and L.R. Snyder, Troubleshooting LC Systems: A Comprehensive Approach to Troubleshooting LC Equipment and Separations (Humana Press, Clifton, New Jersey, 1989).

(3) D.R. Stoll and T. Taylor, LCGC North Am. 38(7), 379–384 (2020).

(4) J.W. Dolan, LCGC North Am. 22(5), 430–435 (2004).

Dwight R. Stoll is the editor of “LC Troubleshooting.” Stoll is a professor and the co-chair of chemistry at Gustavus Adolphus College in St. Peter, Minnesota. His primary research focus is on the development of 2D-LC for both targeted and untargeted analyses. He has authored or coauthored more than 60 peer-reviewed publications and four book chapters in separation science and more than 100 conference presentations. He is also a member of LCGC’s editorial advisory board. Direct correspondence to: LCGCedit@mmhgroup.com