[Server-devel] School networks and electrical equipment damage

[John Watlington]

On Jun 6, 2013, at 3:58 PM, Daniel Drake wrote:

> Those of us familiar with setting up school networks (server + switch
> + APs) in some of our deployments will be familiar with  the
> occasional loss of hardware, due to surges in the low quality
> electrical supply or whatever, even when the system is protected by a
> cheap UPS which supposedly offers some protection.
> 
> This has often been the case in Nicaragua, so the group is now buying
> more expensive UPSes, PoE switches, and PoE access points for new
> schools. This means that the server and switch are connected to mains
> power via a UPS which hopefully protects them, and none of the APs are
> connected directly to the mains (instead they get Power over Ethernet)
> which hopefully offers some isolation from bad electrical conditions.
> 
> This equipment is expensive, especially in places like Nicaragua where
> lots of import taxes are applied. But the hope is that the investment
> pays off in that the equipment doesn't get zapped.
> 
> However, one week after deploying this equipment in the first school,
> we are left with a server that doesn't boot, 3 out of 4 access points
> broken with a nice burning electronics smell, and a broken switch with
> a lot of visible damage to the electronics.
> 
> And the most surprising thing - we had not even turned on the network
> yet, pending some electrical work. Everything was connected up except
> one crucial link - the UPS was not plugged into mains power. So all of
> this damage happened without any of the devices having a connection to
> the mains.
> 
> Connectivity-wise, the setup was:
> WAN: Phone line - ADSL modem - XS
> LAN: XS - Switch - 4 APs
> 
> And power connections: the XS, ADSL modem and switch were connected to
> the UPS. The APs were connected to the switch over ethernet for both
> power and data. Again, since the battery was not connected to mains
> power, none of the devices had a power source.
> 
> The connectivity engineer's best bet is that a lightening bolt landed
> at the school or nearby, and that this caused a power surge on the
> phone line. This surge passed through the ADSL modem, server, switch,
> and 4 APs, destroying everything in its path (except 1 AP that was
> connected over a longer cable than the rest).

This was my most likely hypothesis as well.
I believe the damage would have been less had the UPS
actually been plugged in, but most probably have their input
protected, not their outputs!

> I figured this is a story worth sharing, for any other projects
> considering splashing out on more expensive equipment...

> Also, I'm wondering if anyone has any advice/experience here. Would
> others expect this more expensive setup to be more resilient to bad
> electrical conditions than a cheaper setup - will the investment pay
> off?

Cat5 Ethernet transformers generally provide 1.5 kV of isolation
(at each end) but PoE breaks that.

I believe the actual protection provided by the UPS can vary widely.
They are usually required in situations like yours to protect the hard
drivers, not as much for power line protection.   I would use a periodically
replaced surge protector before it and maybe a surge protected power strip
after it.

> I figure that the case of a lightening bolt might be a bit extreme,
> but electrical storms are a nightly occurance here almost daily during
> the 6 month rainy season.

Nothing will protect against a direct lightning strike of the wire.
The more common case is a strike near the wire or the central office which
can induce still induce many kV of surge on the lines.   If electrical storms
are common you need to take precautions.

I hope all the network cabling is indoors ?   If it is only partially so,
consider something like:
http://www.l-com.com/surge-protector-indoor-4-port-med-power-10-100-base-t-cat5-lightning-surge-protector

> I have seen that some UPSs (unfortunately not these ones) allow a
> phone line to be passed through them, supposedly offering some
> protection. Would such a system protect against a lightening bolt,
> assuming thats what happened here?

Probably not.

Primary protection on the phone line should be an gapped carbon block
or gas tube protector at the entrance to the building between each line of
the phone pair and a good ground.    These protect against higher
energy surges, and generally kick in at over 1 kV.

I would suggest using primary protection that includes secondary protection,
such as you see in:
http://www.l-com.com/surge-protector-outdoor-high-power-telephone-dsl-lightning-surge-protector-screw-terminals

You could also loop it through the UPS at this point for redundancy
(but it MUST remain plugged in to provide protection --- it not, it makes
things worse!)

Then there is the protection in the modem itself, which should be able
to handle the remaining surge.   They are so common in telephony as to be
required --- but primary protection and proper grounding is always assumed.

Sorry to hear about your misfortunes,
wad