Disaster Recovery: Did you consider your branch offices?
The Northeast is still drying out after record rainfall that caused major floods in several urban areas. For the most part, large corporate data centers were spared; however, there were a number of large corporation branch offices that were not.
These branch offices incurred substantial flood damage and consequently a loss of business and revenue. In my article last September, "Disaster Recovery: Are you Prepared", I focused on the large Corporate HQ and Data Center environments where a disaster had the potential to impact the entire enterprise. An Enterprise Disaster Recovery Plan should indeed be anchored by the Corporate Headquarters, but most certainly must extend to the most remote locations of the Corporation regardless of the location.
This is especially true for those enterprises in the services and merchandising sector, whose services and merchandise will be in high demand after a major storm, flood, or other natural disaster. This will not only bring more short term revenue to the corporation, but instill an image of strength and reliability to the consumers and the community which the business serves.
The Key to Success: Customer Notification Systems
From an Information Technology perspective, corporate headquarters should have robust customer notification systems in place which can drill down to the location level and then set in place notifications to all customers that the impacted location may serve. This could be automated e-mail systems, localized web page announcements, or a number of other notifications depending on the business and the customer. The key is having the information available on the spur of the moment and having a precise plan in place which can be executed flawlessly on demand.
Technology Infrastructure Team must be ready for Rapid Deployment
The technology infrastructure team must have a solid plan in place to support the remote branch offices as well. This plan must include stocking customer facing point of sale systems and network components which could be casualties of fire, wind, and water damage. Another vital component of the plan is to have personnel on call 7 days a week 24 hours a day ready to travel to any location at a moments notice.
In my case, my team is on a rotational call basis which means they are usually on call one week a month. During the week the engineers are on call, they are expected to be on the next flight out if we have a disaster at any one of our locations. Most recently during the northeastern floods I had feet on the ground at two locations as soon as the flood waters subsided. In both situations my team deployed emergency infrastructure equipment and had the locations back on the network and operational as soon as the authorities permitted.
Speaking of the local authorities, it is essential that the local business operations managers maintain a good relationship with local officials. In most cases the local authorities will ban access to impacted areas. This is primarily to prevent looting and injury; however, if your business has a legitimate disaster recovery plan in place and a good relationship with local authorities, you can be assured your recovery teams will be permitted access as soon as it is safe.
Wide Area Network Redundancy
Today it is not uncommon for any point of sale system to ultimately rely on accessing the server or mainframe at HQ. As a result, the wide area network connections at the remote branch offices must be considered and accounted for in your disaster recovery plan. While dial-up is an old standby, this mode of backup is quickly becoming obsolete due to performance issues with throughput as well as the reliability of the wired network to the premises during a disaster such as a storm or flood.
As such, wireless technology must be considered and included in any disaster recovery plan for your branch offices. As a minimum each location should have an emergency POS system installed on laptop with a wireless EVDO card installed. A standby system such as this has been a savior to more than one of my branch offices on numerous occasions.
Fixed Wireless and WiMAX
However, with the deployment of WiMAX and fixed wireless communications taking hold today, wide area network engineers should be designing redundant wireless connections to their branch offices. I have accomplished this feat at all of my offices in the Toronto, Canada area where fixed wireless is widely available. This technology has proven to be very effective and has already saved the day a couple of times when an errant backhoe cut some fiber cables in the area.
Unfortunately in the USA, the Regional Bell Operating Companies (RBOCS) are lax in delivering this capability, but it appears they have seen the light... Or maybe it is the opportunity for additional revenue. As a matter of fact, I just completed a 120 WiMAX trial with one of my offices in Atlanta. The trial was outstanding to say the least, and exceeded the reliability and throughput we are accustomed to in todays wired network.
Most recently, Bell South has announced plans to launch fixed wireless services encompassing five new markets in Mississippi, Tennessee, South Carolina, and Florida. It is only a matter of time before the other RBOCS jump on the bandwagon. So if you have offices in any of these areas, you should be thinking about implementing redundant wireless connections to your branch offices.
In summary, it is essential that your disaster recovery plan be well documented and takes into consideration the all of the disasters that can befall to your branch offices. The branch offices is where the rubber meets the road and where the revenue is generated that keeps the corporation afloat and pays our salaries, so it is essential that these branches are kept operational or restored to operation rapidly after any disaster.
I could write a book on what you should do, however, in the final analysis each business has different requirements, which only you and business operations can identify and develop a blueprint on how to mitigate the impact of a disaster.
Last, but not least, you must constantly be on the lookout for new technologies such as fixed wireless that can make your existing WAN infrastructure even more resilient than it is today.
Webcast: Enterprise Information Protection--Best Practices and Policies for Internet Security