Learning the Art of Instrument Troubleshooting

This repair required 5 boards, a secret trip by Roger, and a team of colleagues to fix, 2012

. 

 The lab has a bright red hat from the University of Arkansas--affectionately called The Pig Hat, which I wear when solving a difficult mass spectrometer problem. Call it a totem, sometimes it seems it might actually help me to think a problem through. I started out simply.

“Show it who’s boss,” Tom Hoering would say, after walking into the lab and seeing my worried face staring at a mass spectrometer that wasn’t working as it should. He shouldered the burden of troubleshooting during my early career and he enjoyed that role. As new techniques came into the lab,  however, I upped my troubleshooting game. It’s vital to having a productive career.

 

“Isolate the variables,” Thermo engineer Frank Trensch would say when you called him frantic to learn what was troubling your mass spec. Thermo-Finnigan’s isotope engineering team has included many helpful, but sometimes over worked technical support folks. Roger Husted was my favorite—often diffident but humble and always knowledgeable. Bill Holmes, current gas chromatography ISOLINK guru is a sunny fellow—always upbeat, polite—and organized, helpful. I worked with this group over decades to keep the lab going strong.

 

Learning to troubleshoot an isotope system—the mass spectrometer, automated analyzers, or preparation systems—is an art learned over many years. Read the email traffic on the isotope geochemistry listserve “Isogeochem” and you’ll learn about the nuts and bolts of keeping a sophisticated isotope lab running in tip-top shape. Caltech professor John Eiler once commented at AGU that essentially, if you don’t understand how your instruments work, you’ll never get too far. With a grain of salt, he’s right.

 

Here’s some of advice to those of you just starting out and figuring out how to know what to do when your system goes down—which it will—eventually.

 

#1 Learn how things work when they are working as they should.

 

This might sound obvious, but it’s important. When your new instruments are installed in your lab by an engineer, spend as much time as you can learning the ropes. When the engineer leaves (and you may feel a bit of angst), jump right in now. Don't wait. Figure out how the software works, read through the manuals (maybe even print them), analyze standards, make new methods, and establish a baseline of knowledge.

 

Get familiar with the vacuum system, the ion source, the electrical flow, your power source, and the physics of mass spectrometry. Have a look at Nier’s original papers, read the early works by Craig and others from the 1950s and ‘60s. In other words, do your homework. Please—don’t complain about having to weigh plant samples into tin boats.

 

#2 Get proficient with Swagelock fittings and glass capillary fittings.

 

My first time using Swagelock metal fittings was a disaster, as I had them all put together backwards! A poorly made connection will leak and cause problems quickly. For capillary tubing, learn how to cut it, how to assemble those seemingly backwards fittings now—before you need to do them on your system.

 

Swagelock fittings. Doug Rumble and my lab, 2009

 

#3 Buy a good, complete set of tools.

 

For me, shopping for clothes and shoes is something I avoided unless absolutely necessary. But, I loved strolling into Sears or a hardware store and picking out shiny new tools—wrenches, screwdrivers, nut drivers, socket wrench, hammer, and drills. Go online and purchase specialty tools for your systems—tubing cutters, diamond scribes, files, tiny drills, etc.

 

#4 Purchase a leak detector and a flow meter.

 

Don’t go cheap here and don’t wait until you need to use one to buy one. Buy these now and learn how to use them. Even if you are not using continuous flow techniques, they come in handy.

 

#5 Purchase and learn how to use a digital multimeter to measure current, voltage etc.

 

I’m guilty of not following this one as much as I should have. These meters can save you a lot of time when you are tracking down electrical problems. Don’t just buy one and stick it in a drawer; actually figure out how to use it.

 

#6 Figure out how the software works and backup the hard drive—now!

 

As a Mac user, I am nearly helpless with the PC software and computers. That said, spend a couple months figuring out a crashed hard drive once and you’ll be glad you at least learned the basics.

 

#7 Keep a good supply of spare parts—ferrules, nuts, fittings, etc.

 

Waiting for spares costs time—and time is “money” even in labs where analyses go on without funds exchanging hands. My lab has always served as the “stock room” for others, who are hesitant to make investments in spare parts. I encourage you to follow my lead—better to have than to have not.

 

#8 When the first sign of an analytical problem surfaces, don’t ignore it.

 

This also might sound obvious but there is the inclination to think a problem might go away on it’s own or after running 10-20 more standards. My favorite is getting notified that the filament light is “out”, but it just might be a burned out LED bulb rather than a filament that needs repairing. I have never had the light bulb burn out, just open filaments. Get ready to isolate the variables and show it who’s boss.

Noreen and Marilyn with the wrenches.

 

 

#9 Remember—it’s only a machine. You can fix it.

 

And if you really can’t fix it, you can start to figure what is wrong. 

 

 

·      When did the problem first show up?

·      What was being analyzed at the time?

·      Who was the analyst?

·      What are the symptoms?

·      Have they been seen before?

·      Is it a steady problem or an intermittent one?

·      Have you run diagnostic tests?

·      Has it been described in the manual or on Isogeochem?

 

Before replacing things that might be broken, consider all the options. Pick the likeliest one, essentially isolating the variable. Don’t start making many changes now, because you may never learn which one was actually the problem.

 

If you have another person you can discuss things with, now is the time. Doug Rumble and I discussed all of our major repairs after Tom Hoering died. It was helpful to put things into words and get a 2^nd opinion. This is a good time to contact your mass spec’s service engineers and see what they think. In my experience, these folks have seen almost anything. Armed with data and not just “it isn’t working”, you often will get confirmation or other great advice.

 

Roxane fixing an ion source.

There may be some aspects of fixing a system that are not your strong suit. I confess I never independently cleaned an ion source. At this time, well, I couldn’t even begin, but even when my hands were working normally, I didn’t have the skill to do that job right. I partnered with those who could or trained others who had the skills that small, finicky work demands. Some people actually enjoy cleaning the source!

 

#10 Don’t give up. Be patient.

 

Get the schematics of the system. I can sort of read electrical schematics, if I spend enough time staring at them. My degrees in biology and botany did not require me to learn engineering diagrams. Fortunately for me, they are abstract and with a little knowledge you can learn a lot. Eventually, you will get there. Recently, we had a hard drive fail on a triple quad mass spectrometer that was not backed up! (My bad.) It wasn’t a simple fix, especially for those of us who aren’t familiar with PCs. Postdoc Jon Nye finally figured out with Thermo service techs about IP addresses and pinging. The attention is in the details.

 

Repairing a combustion reactor,

Those with gas chromatographic systems will find that their wrenches are never long off the bench before being brought back out. I have found that having a system work >90% of the time is possible, but you can’t be lax.

 

#11 When you’ve fixed the problem, write down your journey so the next time you’ll be prepared.

 

Each time you successfully troubleshoot your instruments it’s a great achievement. Every time you figure out the problem, take the steps to fix it, is a learning experience. Isogeochem’s emails are the perfect testimony highlighting scientists’ dedication to troubleshooting, understanding, and going onwards. It might not get you tenure, but an isotope system that doesn’t produce data, won’t either.  Time to celebrate!

 

24v/5V power supply failed and fixed. All green lights on--our electronic tech fixed it!