Resolving Most Complex Technical Breakdowns in Manufacturing

Last monday, we lost a drive at our facility in Fullerton. This post is dedicated to covering, reflecting and discussing how the whole process went and what could have done better or differently.

The Breakdown Information – Emerson FX 230 Drive

Emerson FX-445 Drive
Although I can’t go into too much detail, I can definitely cover the technology. We have a case packing unit which is about 25 years old at our facility. It has both a PLC 5 (RSLogix 5) and a Micro Logix (RSLogix 5000). Both of these processors are used for I/O and communicate between each other through discrete Input / Output cards. This was built before I joined the company and I was told that the upgrade was “partially” completed to address 4 / 5 drives on the machine.

What broke down is the FX-455 Drive which controls one of the critical servo motors. I received a call on Monday morning and had to drive in to help troubleshoot. The drive, which is extremely old, was displaying an “F” code and wouldn’t complete the full range of motion required.

Diagnostics & First Steps

What made this case particularly interesting is that we had no knowledge on how to work or even connect to this drive. It also has proprietary Emerson software which we had never previously used. Replacing the drive was no easy task as we needed to copy everything from the drive that’s in the machine into a new unit.

As I arrived on site, my first actions were to see the drive myself and understand better what exactly it is doing. I was shown the drive by the tech. who had been on site since 5 AM. Indeed, the drive would fault out on power-up & initialization of the routine. Digging up a manual for the drive, I was able to find “some” troubleshooting steps in the appropriate section

  1. Low bridge AC input voltage and/or insufficient KVA ratings
  2. for external transformer
  3. Accel ramp times too short
  4. Load creating too much friction
  5. Gear reducer oil level too low for high speeds
  6. Wrong type of gear reducer oil (too thick)

Below the above “diagnostics” I also found that the problem can be with the gearbox, load or the AC voltage. Based on what I’ve seen in the past, the problem must have been mechanical. We would decouple this motor from the load and the problem would be solved.

Mechanical Troubleshooting

Before faulting out, the motor would “jerk”. It was attempting a move and faulting out right after. It produced the same results in normal mode and in jog mode. After decoupling the motor from the load, to my biggest surprise, absolutely nothing changed. I was extremely puzzled as the motor which now only had the shaft sticking out would slightly move then fault out the drive.

Resolving the I/O in RSLogix 5

For those of you who have worked with RSLogix, you will know that understanding the I/O points is crucial for electrical troubleshooting. In this case, I had a vague idea of what I was looking for and the location of the logic responsible for the drive. Once again, the PLC was sending about 10 discrete outputs to the drive and receiving the same amount back as inputs on another card. As mentioned before, the drive itself had a program which would determine the position, acceleration, deceleration and parameters of the drive; completely different from a Kinetix 6200 drive I’m used to working with.

After spending some time on the software, I was able to figure out all the appropriate tags. With the help of the tech., we traced back all the wires in the panel. I was finaly able to determine where exactly the drive is “breaking”. By bypassing one of the routines, the drive would reset into an operational state (Indicated by the letter “E” on the front panel). Although not extremely helpful to fixing the drive, we now knew exactly at which point or command we were receiving the fault.

The Hardware Swaps

The typical “brainless” approach to solving most manufacturing hardware problems is to narrow the problem down to a device and swap it out. Very little cost is associated with the device versus how much is lost due to downtime. It’s always easier to replace the part than to take the time to troubleshoot further for repairs.

That being said, due to the indication of the fault code, we decided to swap out the transformer feeding the drive. It stepped down the voltage from 480V to 230V. Although I was able to measure the right voltage, I didn’t have a reliable meeter or scope to confirm that we weren’t dropping the voltage in run, thus faulting out the drive.

We remained positive as the transformer was brought out from the warehouse and swapped on the machine. The process didn’t take too long and we were able to power up once again. For the second time, I store at the same letter “F” as it appeared on the drive LED indicator. The issue wasn’t the transformer.

Board Level Troubleshooting & Replacement

In the four years I’ve spent in manufacturing, I’ve never had to replace a board inside of a particular component. It simply wasn’t needed and we never had enough information to perform such an operation. Very frequently, manufacturers would recommend never to open their devices and service them in house as it will void the warranty.

Rising the Loss of Hardware

The other worry that came to my mind was that by opening the drive and playing with the electronics inside, we risked damaging the board and software which we didn’t have saved anywhere. The fact that we work in a cold & humid environment doesn’t help protect the equipment from potential damage.

Despite the fear, my decision was to take another drive we had at the plant and attempt to change out the power circuit since the error code seemed to point that way. I felt like a surgeon removing the control board from one drive and placing it in another. I knew that if the slightest mistake was made, a pin was damaged or I had any static electricity discharge, we would lose one drive or the other.

Very questionable results

Although the drive appeared to “work” after the board swap, it would not position itself correctly. After trying everything we could on the mechanical side, we made the biggest mistake we could have at that point: powering down to troubleshoot the mechanical assembly. Although I understood what happened after, the pressure of getting the unit running, I thought we could fix our issue by realigning the drive with the mechanical assembly.

If you haven’t guessed what had happened, the battery which was used for the memory of the drive had ran out. The meant that we lost all of the software and configuration on the drive. As we powered the drive, it was no longer responding to the I/O, jog or move commands. It was just sitting there in an Enable state awaiting instructions.

MS DOS Emulator

Earlier that morning, I spoke to our IT admin and asked him for a 32 bit Windows XP machine. The reason was that the drive was running on software built in 1992. Yes, you read that correctly; it was released when I was 3 years old. They had absolutely no support for Windows 7 and said that “it might work”, but we can’t guarantee anything.

The first question the IT administrator asked me was “how did you connect to the drive before?”. The expression on his face was priceless when I told him that I never connected and got the information from the manual that same morning. After the shock settled, he said that he might be able to come up with a Windows 7 machine, but it will have to be a desktop as they didn’t release “second” laptops to people. I gave him the go ahead and said that we can definitely try and see what happens.

As I looked at the softwareless drive, I realized how bad the situation really was. I was looking at a drive that was 30 years old, had no previous experience with it, had no software to connect to it or the right computer. The only piece of comfort was the passage about the software requirements for the drive:
PCX Emerson Software
Once again, your eyes aren’t deceiving you, the software came on floppy diskettes.

Putting it all together

The IT admin brought out an old computer. It has the serial port I needed and Windows 7 installed on it. The software, which I downloaded from the legacy Emerson site did not work on that machine. I was back to square 1.

As I discussed the problem with a contractor, his suggestion was to see if it worked on a DOS emulator. After a brief search, I found DosBox. It’s a DOS emulator used for running very old games. To my big surprise, it only took me about 10 minutes to learn how to use the emulator and figure out how to mount a local drive into it; the process isn’t extremely straight forward, but would take too long to go over. The next challenge was connecting to the actual drive.

I’ve worked with computer hardware before and knew that we may fall into issues with regards to the ports we are using. In other words, the COM1 port on the PC could be there, but not be enabled or working properly due to a mismatched driver. I decided to read a forum post which payed off big time. I found a command which was necessary for the DOS Emulator to be able to work with the computer’s COM PORT. It remained in my memory after all the times I typed it in:
serial1=directserial realport:com1

Hardware Serial Connections

For those of you who have worked with industrial software, will know that things don’t just work. You need to figure out if the cable is straight trough, crossover or a completely different pinout from what the protocol is calling for. For the next hour, I spent playing with different types of serial cables modifying them along the way. At the end, a cable with some wire nuts did the trick! It wasn’t pretty, but we had established communication.

As we reach 2000 words, there will be a Part 2 to this story…

Cheers,
Vlad