Major Network Failure Between Miami & Puerto Rico

ISP's | 12/04/2006 | Various

[cll]

Notice posted at PR host regarding major failure:

Dear [ISP] Customer:

This is to inform you of the events of Sunday, December 3, 2006 which resulted in a [ISP] network outage.

At approximately 10am on Sunday, December 3, a major network event occurred whereby all circuits from Puerto Rico to Miami reported failures. After extensive research to troubleshoot this error, a major undersea fiber cut was found. The fiber cut appears to be in the waters off the coast of Punta Cana [Dominican Republic].

[ISP's] main redundancy, which consists of routes on alternative fiber paths, had been scheduled by our partners for maintenance repairs. Therefore, our main redundancy circuits had been temporarily moved to the same cable as the primary circuits for this maintenance window.

[ISP] maintains agreements with local providers to exchange local internet traffic on-island and to route all traffic via these providers in the event of a major network event. Unfortunately, this fiber cut has affected almost all providers in Puerto Rico. [ISP] was able to restore service by 10pm on December 3, 2006. However, bandwidth is significantly limited for all providers on the island, as we are all sharing resources.

The current prognosis is as follows:

1. We expect that bandwidth will increase as the day passes; we are hopeful that we will be able to add more capacity via new alternative routes. 2. The maintenance and repairs of our main redundant circuit in the Gulf of Mexico has been accelerated by our partners and should be completed sometime by early tomorrow morning (Tuesday, December 5). Full service should be restored by mid-morning on Tuesday, December 5. 3. In January 2007, a new cable will be put into service via the US Virgin Islands. This cable will offer additional redundancies and alternatives for the future.

The cause of this failure has not been determined at this time - whether it was negligence by a partner, an Act of God, or other possible cause. We will keep you posted on this.

[cll]

This sucks. We have not felt so isolated since a date I won't dare mention. Curiously, Venezuela, 500 miles south of here, had a failure yesterday, election day.

[cll]

Puerto Rico ping. In case you were wondering.

Please FreepMail me if you want on or off this list.

[Jedi Master Pikachu]

Does a new cable have to get put in place?

[cll]

News are sketchy. Actually, this is the first solid news we've gotten since yesterday.

I am not in the know but I would think that if it is a fiber optic cable, it would have to be replaced.

Right now we seem to be down 80% in capacity. I have been able to log on only a few times today.

[taxesareforever]

or other possible cause.

You really mean to say "sabotage", right?

[Jedi Master Pikachu]

Appreciated.

[cll]

"You really mean to say "sabotage", right?"

That's from the statement by one of my ISP's. But it looks like a big coincidence that Venezuela also reported a major failure yesterday.

[NonValueAdded]

I question the timing.

[ProtectOurFreedom]

Nope. They pull them up and repair them.

[ProtectOurFreedom]

Here's some info from a few years ago. Can you imagine fishing for a cable 9,400 meters underwater?

Alcatel breaks world record in deep-sea cable repairs - Undersea
Fiber Optics Weekly Update, June 27, 2003

Alcatel has successfully completed a fiber optic cable repair in water depths close to 9,400 meters -- breaking the world record for deep-sea cable repair operations. The Alcatel operated cable ship, C.S. Heimdal, completed the fiber optic cable repair approximately 700 km off the coast of Japan on a transpacific cable system connecting Japan to the U.S. continent. Because of the extreme water depths involved, Alcatel created a special grapnel to avoid the cable breaking under its own weight as the cable was raised from the seabed for repair. Based in Japan and fully mobilized with remotely operated vehicles, the C.S. Heimdal vessel enables Alcatel to help its customers respond to potential cable outages within 12 hours.

[ASOC]

Cable "burn thru" is one possible cause of the failure. Most long US undersea fiber cables are DC powered using the ocean as a return path for the DC power.

If the cable sheath is damaged the cable, in essence, shorts to the ocean. which can cause the copper conductor in the cable to burn away or dissolve - like electroplating in reverse, for lack of a better description.

They fish the cable up, splice on a new section of cable and run along till they find the next 'whole' section, another splice, and testing to confirm the repair.

Damage can be caused by fishing gear - the cable are normally found on charts - or by anchors, seismic events or chafing from tidal or current moving the cable across the seabed. More can be found here with links to photos.

[xrp]

JAWS not available for comment.

[cll]

Dang. So it's going to be a while.

Thanks to both for the info.

[Jedi Master Pikachu]

Appreciated.

[FReepaholic]

...If the cable sheath is damaged the cable, in essence, shorts to the ocean....

Fascinating. Thanks for the info. FReepers are the best!

[ASOC]

LOL

Just a lucky break, I happened to had (at one time) responsibitly for two major US FO cables from AK to CONUS. Fishing gear was our #1 worry followed by, of all things, earthquakes of the coast of ORE.

Now, I just use the 'net and let someone else lose sleep over cable cuts..... it *is* way cool technology tho, esp the erbium-doped parametric fiber optic amplifiers.

[cll]

From the link in post # 11:

The first small diameter fiberoptic cables were damaged by fishing boats with drag nets and by sharks chewing on them on the ocean floor...

[AZLiberty]

Cable "burn thru" is one possible cause of the failure. Most long US undersea fiber cables are DC powered using the ocean as a return path for the DC power.

This doesn't make sense to me. Fiber-optic cables are passive: you just shine light down them and pick it up at the other end. Or is the power for repeaters along the cable?

[livius]

Curiously, Venezuela, 500 miles south of here, had a failure yesterday, election day.

Strange indeed.

[ASOC]

The cable is is (from inside to out) 4 fiber strands
a Teflon wrap
a copper 'pipe' to carry voltage
another layer of Teflon. In places the cable may be unarmored and about 5/8 of an inch thick.

Armored cable can be 2 or 3 inches in diameter.

The amplifiers, which sit about every 100kM or so, are used to boost signal levels, get their power from the coper in the cable.

See my earlier post for erbium doped amplifiers - they are really quantum state energy pumps - cool sorta new (1999) tech -- also used on land based systems as well.

An interesting sidenote, since the cables from AK to CONUS are DC powered, the copper inside the cable acts almost like a 3,000 kM antenna.

During magnetic storms, system feed voltages can vary by about 100 VDC as the earths magnetic fields 'cut' the antenna and induce voltage, acting like a big generator.