Fire in a server roon. Smoke/soot damage

[sram]

Unfortunately, we had a fire in a big server room. The AC was the culprit. There were like 20 servers in there. The servers didn't actually get burned, but was only affected by the thick black smoke(Soot). Parts of the servers were already starting to rust because of oxidization.
Internal parts are all covered with a black layer of soot.

How would you assess the situation? The servers are no longer usable??

Thick smoke I think will cause a short-circuit, right? If this is the case, then all electronics inside will be just bad. It will be just like dipping the boards in water. right?

What would be the best course of action?

The server room is currently offline(Which is fine because the system they should be running isn't fully deployed yet). We would expect reliable operation from the server room when they start to work, but at the same time we don't to buy replacement servers if the smoke damaged ones can be made to work properly.

I can provide photos if needed once I get them.

Please feel free to ask questions.


Thanks in advance

 

[Nigel Goodwin]

Presumably the servers need a HIGH degree of reliability?, replace EVERYTHING!!

 

[ronsimpson]

I found the fire extinguisher chemicals did most of the damage.
I think the fans will not work or will not work for long.
The "dirt" in the parts (like heat sinks) will make the parts run hotter than normal.

If the information on the hard drives is critical, I think they will work in new computers.
To test the computers, I would take a air compressor to the inside of the computers. Maybe a soft brush to help remove the dirt/soot.
Watch the fans. They may not last long.
Worth a try! There maybe no damage. Do you feel lucky?
If the computers are critical then you need new ones.

 

[sram]

Presumably the servers need a HIGH degree of reliability?, replace EVERYTHING!!

OMG Nigel, you are still breathing! I think you are most helpful in the whole world when it comes to online forums. How is your eyesight now?

Anyway, you are right. The servers need to be reliable of course, so "replace everything" is the first thing that comes to the mind but somebody's boss want a more informative answer!

 

[ronv]

If you have more time than money you could try disassembly and wash in a dishwasher with this stuff.
https://www.ebay.com/itm/Superior-S...er-/121477399084?_trksid=p2141725.m3641.l6368
You will need to be careful of switches and fans.
Make sure everything is really dry.

 

[granddad]

sram Can you tell us what was wrong with the AC , might be useful to other IT managers....

 

[sram]

I found the fire extinguisher chemicals did most of the damage.
I think the fans will not work or will not work for long.
The "dirt" in the parts (like heat sinks) will make the parts run hotter than normal.

If the information on the hard drives is critical, I think they will work in new computers.
To test the computers, I would take a air compressor to the inside of the computers. Maybe a soft brush to help remove the dirt/soot.
Watch the fans. They may not last long.
Worth a try! There maybe no damage. Do you feel lucky?
If the computers are critical then you need new ones.

We will of course try to recover every bit we can from the hard drives. I'm not usually lucky, but it is worth a try like you said. We may get them--or at least some of them-- to work, but they can't be guaranteed for reliable operation. That's the biggest problem I guess.

If you have more time than money you could try disassembly and wash in a dishwasher with this stuff.
https://www.ebay.com/itm/Superior-S...er-/121477399084?_trksid=p2141725.m3641.l6368
You will need to be careful of switches and fans.
Make sure everything is really dry.

Haven't seen something like this before. Does it really work? Anything similar that works like a spray?

sram Can you tell us what was wrong with the AC , might be useful to other IT managers....

I don't have the details, I'll have to look at the firemen report and come back here.

 

[atferrari]

About 2 years ago, I witnessed firsthand the fire in a vessel that affected most of her cargo. Later, I participated in all the subsequent inspections to parts that were still on board during the incident.

Besides many things, the control room was extensively affected by soot. The technicians refused to attempt the recovery and rejected the equipment.

 

[BobW]

We will of course try to recover every bit we can from the hard drives. I'm not usually lucky, but it is worth a try like you said. We may get them--or at least some of them-- to work, but they can't be guaranteed for reliable operation. That's the biggest problem I guess.

Hard drives are sealed units, and unless they were affected by the heat from the fire, the data should be recoverable from them.

When you say the fire was caused by the AC, did you mean an air conditioner failure, or the AC power?

 

[MrAl]

Hi,

Long time ago i used to help restore burnt out houses so i got a lot of stuff that had what we called "fire damage". It wasnt really burnt, just mostly blackened from the smoke, so it was mostly smoke damage not fire damage. I remember a nice audio amplifier that had part of the plastic melted and smoke damage. It still worked though and still worked for years after that too.
So it all depends on what actually happened to the units, how close to the flame, how much smoke damage, what parts the smoke damage actually got to.. Some might still work while others might not work at all. Some might work for years others might fail after a week.
This is a very variable area because it is too hard to evaluate what actually happened to each and every unit. Also, where they running when this happened...then that would probably have sucked in more smoke than if they were just sitting there. Also depends on the natural air flow when this happened.

Probably the only way to be sure is to test each unit individually and make an assessment. Keep some new backup units ready to go in case any give out. That's if any still work that is. If none work during testing then your course of action is clear.

 

[tunedwolf]

Get your hands on a toner vacuum, the kind that photocopier engineers use, hoover everything on the outside casings and work your way in, disturbing as little as you can at each stage, only after you can see no dust/ soot use a small soft bristle paint brush along with the vac to remove any last little particles. DO NOT USE COMPRESSED AIR to blow anything, all you will do is force carbon deposits under devices that you can not get at to vacuum away.

Once you have removed all the dust/ soot, fire up each server and immediately back up your data. Of course your back-up scheme should already have disaster planning built in, so worst case scenario is you need to restore from your latest back-up set. After you have backed up your data, perform a complete data wipe. Sell the equipment on to someone setting up a test lab or give it to your admin staff to play out configuration scenarios on and the like, the equipment can no longer be warranted and any SLA's you have will no longer be honoured by your vendors. Replace the equipment re-install and restore your data back-ups.

Do you want to be the one to explain to the CEO that his network is down because that smoke damaged DC or file server finally failed? No? I think you know what to recommend to the boss's boss, put it in writing so there's no excuses, doubt or finger pointing later

 

[KeepItSimpleStupid]

I'd probably let insurance handle it. If there is any data that needs to be recovered I'd remove the drive and recover it on a good clean system. The drives are sealed but static damage is still possible trying to clean it. A HEPA anti-static vacuum would be mandatory. e.g. https://www.amazon.com/Metro-Vacuum-DV3ESD1-DataVac-Anti-Static/dp/B000RMQJBK You could also send the drives out to be recovered. That may be the safest option.

You will likely have to wipe or destroy the drive if you dispose of the machines.

The soot will damage the mechanical parts like the fan. The soot will make the systems run hotter, so I'd remove the HD and clean before powering it back up.

Tektronix did actually wash their scopes in a dishwasher and there is a specific procedure to follow, particularly drying. Batteries would have to be removed. Not sure i would recommend it. I could probably search for the procedure.

There is some info here: https://www.antiqueradios.com/forums/viewtopic.php?t=133594 Also as suggested you have to be careful using air. Canned air is pretty clean. but your super duper air compressor may have water and/or oil in the air.

Freon TF in an ultrasonic would work very nicely, but that's environmentally banned now.

 

[ClydeCrashKop]

I agree with Tunedwolf on the vacuuming without touching. Then "Spray Nine" will usually remove carbon soot without scrubbing. I don't know how you would keep it out of relays, etc. though.

**broken link removed**

 

[rumpfy]

SRAM,
I dont know which country you are in.
Generally, the cable industry has been developing halogen free, low smoke cables for a number of years.
The development of such cable was brought about because of fires doing serious damage. The conventional insulation and sheath material for internal cables was PVC. Unfortunately, this material releases copious amounts of chlorine. With water from fire extinguishers etc., the chlorine combines with the water to produce hydrochloric acid. If the building is a concrete/steel design, the long term damage is serious.
You say the fire started in the AC system. Usually this sort of gear is made with cheap cable and wire and is almost guaranteed to be PVC. My approach would be to exercise caution in your assessment, because the cooling fans etc will almost certainly distribute the soot and chlorine byproducts right through the computer gear. failure in the long term is almost guaranteed.
Many companies and organisations are specifying alternative cable constructions. Generally these are based on halogen (chlorine/bromine) polymer blends and/or include aluminium salts. The type of material chosen depends on whether you are dealing with LV or HV power cable or whether you are dealing with ELV control cable or maybe telecommunications/data cable.
The USA, Europe, Australia are switched on to these new designs.
hope this helps.

 

[sram]

Thanks for everybody who contributed to this thread. I'm attaching some images of a switch pulled of one of the racks. This is so that you can have a feel of the damage. I forgot to ask about the specific reason why the AC burned, but I'll have an answer soon.

I think you can clearly see the layer of soot, and the parts that started to rust because of oxidization.

 

[KeepItSimpleStupid]

Wow! Absolute Yuk!

Here's https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CB8QFjAAahUKEwi-hICngt7GAhVTm4gKHU40DM4&url=https://www.intuitiveaccountant.com/training-center/cleaning-up-your-computer-after-a-fire/&ei=PMimVb72KdO2ogTO6LDwDA&usg=AFQjCNFSFB-iJq977_LQVG6kakUy7x9cDQ&sig2=3NwQWErSd-_EUHCMZFx67A&bvm=bv.97949915,d.cGU&cad=rja a little blub beasically reiterating vacuuming, and paying attention to static, doing it quickly and being careful with the use of compressed air.

This https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=6&ved=0CGYQFjAFahUKEwiE6IC7g97GAhWKM4gKHX-QAa4&url=https://www.kb6nu.com/wp-content/uploads/2013/02/tek-wet-washing.pdf&ei=c8mmVcTiBIrnoAT_oIbwCg&usg=AFQjCNER2RiwslYP3RSoqRGwMGvUym12Nw&sig2=iY15L97HNwEJTHvmgwlnOQ&bvm=bv.97949915,d.cGU may be the Tek procedure.

See also: https://www.google.com/url?sa=t&rct...=JIQpWaz6pOhucGrUIAp2Qw&bvm=bv.97949915,d.cGU

 

[tunedwolf]

Poor wee thing, I guess that Cisco multilayer switch will need some love. The good news is, I have actually seen much worse than that being recoverable
Just work your way into the equipment with the vacuum and clean it up as best you can. I'm betting that greater than 90% of the equipment will "fire", ahem, right up

Of course, realistically speaking, none of it can go back into your production network, you know that already, but if working fine after a good clean, I'm sure it will find a happy home in the Admin's play room, or in someones test lab environment. If your boss really wants to be charitable, he could have it cleaned and tested and then gift it to the local community College for training purposes. I'm sure the students would benefit from having some real equipment to work with in the lab, rather than having to use Packet Tracer and GNS3. Your insurance will be covering the cost of replacement in any case, so it's no real cost to your boss other than the cleaning.

 

[Nigel Goodwin]

Like I said way back at the beginning of this thread, replace the lot - these pictures just reinforce that - in fact I'm totally bemused that you've even asked the question in the first place?.

 

[sram]

The total cost of the replacement quotation is about 700,000 $. For somebody who isn't technical, it is hard to justify that over some smoke damage. He was like: Thank God the servers didn't catch fire, let's just clean them and get back to work.

But he should listen to us. We will put it in writing like somebody said here.

 

[BobW]

He was like: Thank God the servers didn't catch fire, let's just clean them and get back to work.

Years ago I did a lot of site inspections to evaluate fire damage. The damage due to fire itself was almost always insignificant compared to the smoke and water damage.