09.11.2018

NCREN Update – Hurricane Florence

FINAL UPDATE:

As of approximately 4 p.m., Thursday, September 20, all NCREN core backbone operations have returned to normal status as our facility in Wilmington is back on utility power. We still are tracking about 18 locations with localized issues. As flood waters recede we will begin the process of inspecting our fiber routes to make sure that none are at risk due to washouts with roads, sink holes, etc.

As always, our NOC staff is available 24/7. If you have any concerns or questions contact the NOC at 1-877-GO-NCREN (877-466-2736).

Thank you for your continued support of NCREN.

UPDATE: September 18, 2018

Hello NCREN Community,

Thankfully, the rain has subsided and our network is starting to get back to normal operations. In addition, many of our customer endpoints are returning on the network as power is repaired in the areas affected by the hurricane.

In further detail, we saw 148 network endpoints go down before and during the storm due to shutting down prior to the event or losing service due to localized power issues. In addition, most of NCREN’s remote regeneration facilities in the east have returned to normal with grid power. As of this writing, we have two facilities on generator power: Wilmington and Pembroke.

Be assured that the NCREN operations and engineering teams are monitoring all of these issues. Once areas are determined to be safe, engineers will dispatch to evaluate and repair all issues under our control.

Here is a list of known issues:

119 of 148 locations that had outages have returned to active status
Two remote regeneration facilities remain on generator power: Wilmington & Pembroke

As always, the NOC staff is monitoring the situation 24/7. MCNC will dispatch when safe and the roads are passable. If you have, any concerns or questions contact the NOC at 1-877-GO-NCREN (877-466-2736).

Thank you,

Todd Broucksou
Senior Director of Constituent Support

UPDATE: September 17, 2018

Hello NCREN Community,

As Florence moves north and out of the state, the after effects continue to cause issues with regional flooding. However, there is some good news with the weather clearing in many areas; the repair of power lines are moving at a faster pace.

The NCREN Network Operation Center staff as well as MCNC’s senior leadership team have been monitoring the network and the weather conditions since the beginning of this event. At this point all core backbone operations are still operating normally with all outages localized to individual site end points.

In further detail, we have seen 147 network end-points go down because of shutting down prior to the event or losing service due to localized power issues. In addition, NCREN’s remote regeneration facilities in the east have started to return to normal with grid power. As of this writing, we have two remaining on generator power: Wilmington and Morehead City. We have been able to reach the Morehead City facility with a portable generator to begin the process of recharging the batteries. The reason the primary generator is offline is still under investigation. Temperatures within the facility have risen but are remaining within NEBS compliance ranges. In addition, IP connectivity from the Bolivia POP continues to be out due to required adjacencies in the area from our MPLS service being out of service; however this is not creating additional outages as the sites that would normally be served are offline.

Be assured that the NCREN Network Operations and Engineering teams are monitoring all of these issues, the weather, and the road conditions. Once determined to be safe, engineers will dispatch to evaluate and repair all issues.

Here is a list of known issues:

147 network endpoints offline since Thursday
51 of the 147 have returned to active status
Two remote regeneration facilities remain on generator power: Wilmington & Morehead City

Thank you,

Todd Broucksou
Senior Director of Constituent Support

UPDATE: September 16, 2018

Hello NCREN Community,

As you already know, Florence has moved into South Carolina from its landfall in the Wilmington area on Friday. However, the slow moving storm and its endless rain continue to cause flooding, power loss, and other related issues in our state. The NCREN Network Operation Center staff as well as MCNC’s senior leadership team have been monitoring the network and the weather conditions since the beginning of this event. At this point all core backbone operations are still operating normally with all outages localized to individual site endpoints.

We have seen 133 network endpoints go down as a result of shutting down prior to the event or losing service due to localized power issues. In addition, NCREN’s remote regeneration facilities in the east have experienced one of more power hits, with four remaining on generator power since Friday evening (Wilmington, Whiteville, Fayetteville, and Bolivia). A fifth, Morehead City, was operating on generator power until early Sunday morning, but it has now gone offline. Equipment at the facility is still operating on DC battery and should for approximately 18 more hours. Temperatures within the facility will rise; however, due to HVAC resources not working while the generator is out. In addition, IP connectivity from the Bolivia POP is out due to required adjacencies in the area from our MPLS service being out of service; however, this is not creating additional outages as the sites that would normally be served were already offline.

Here is a list of known issues;

133 network end-points offline since Thursday
25 returning to active status
Four remote regeneration facilities on generator power; Wilmington, Bolivia, Whiteville, and Fayetteville
One remote regeneration facility on battery power; Morehead City
- Generator repair company has been notified and will dispatch when roads are passable; MCNC maintained portable generator will be dispatched when roads are clear to get site back on-line.

As always, the NOC staff is monitoring the situation 24/7. MCNC will dispatch when safe and the roads are passable. If you have any concerns or questions contact the NOC at 1-877-GO-NCREN (877-466-2736).

Thank you,

Todd Broucksou
Senior Director of Constituent Support

NCREN Storm Emergency Preparedness - Hurricane Florence

September 11, 2018 -As weather conditions related to Hurricane Florence evolve, the NCREN Network Operations Center is closely monitoring NCREN infrastructure and will remain diligent should there be any impact to the NCREN network.

If you are planning to proactively shut down your site as a precautionary measure please let us know. This will help keep our staff from making unnecessary calls. When you have safely restored power, please notify our NOC in the event your service does not restore.

Our NOC will continue to provide updates on any issues that may affect NCREN infrastructure and services through our normal method of distribution and will maintain our high standards to ensure that all problems receive immediate attention.

We have precautionary measures in place, that include:

A 24x7 Network Operations Center, staffed with skilled network analysts and engineers;
Immediate access to senior-level network engineers for escalation, coupled with solid partnerships with our providers;
A fully redundant network backbone, with the ability to reroute traffic, temporarily or for a prolonged period of time;
Colocation in commercial facilities with back-up power to guard against municipal power outages;

Should you wish to reach us for any reason, please contact the NOC by phone at 1-877-GO-NCREN (877-466-2736).

Please stay safe!