Case Study: Upgrading Infrastructure
Summary
This client made a decision to relocate to a new custom built facility housing corporate offices, a banking branch, their headquarters and a state of the art data center. Plans for the data center included significant infrastructure upgrades as well as acquisition of new hardware. The new facility was designed to house all production and non-production equipment as well as room for future infrastructure growth. This represented a significant improvement to the current location where because of space limitations in the data center, services and that most of the equipment was scattered throughout the building.
Project Requirements
The requirement: Relocate 200 servers and devices to the new location over a holiday weekend beginning after the 2 pm close of business day on Saturday and returning the data center to normal operations by Sunday afternoon. Approximately 80% of the devices had to be relocated and fully operational in a twelve hour move window.
Planning Process
Client had a top notch Facilities and IT Team working on the plan to build out the new facility and relocate the devices. Planning for the migration began twelve months prior to the planned move date. Recognizing the need for professional expertise on data center relocation/migration, the Client selected MIGRA to assist their staff in planning and executing the migration. The MIGRA Project Manager led the intensive eight month planning process using a four phase process: (1) Requirements Gathering; (2) Design Planning Process; (3) Implementation Planning and (4) Execution. Planning was completed in the first six months with Day of Move scripting taking place for the next two months. In the first phase, the team documented the Present Method of Operation (PMO) that formed the basis for all the following phases. During the process, MIGRA’s Subject Matter Experts augmented the work of the client’s core team. We developed a detailed inventory, Visio drawings, documented the entire physical Network Infrastructure and the logical application linkages for all devices! Using the documented Present Method of Operation (PMO), the MIGRA planning team developed the recommendations for the Desired Future State (DFS) The Client took advantage of the planning process to identify technology refresh opportunities that would improve the new operating environment.
With the PMO created, the DFS identified the PMO was approved by the client, the combined team moved into the Design phase where the “roadmap” to get from the PMO to the DFS is created. This is where net new equipment needs were identified, changes to the telecommunications architecture were developed and the network within the building designed. Client included other partners in the planning process which contributed to a comprehensive Design Plan which was approved by them.
Once the Design for the DFS was determined, MIGRA then developed a detailed inventory, Visio drawings, documented the desired physical DFS Network Infrastructure and the logical application linkages for all devices as identified through our planning process and design plan. The Implementation Plan included rack layouts and data center cable plans that were outsourced to the client’s network service provider. Based is information we developed a comprehensive list of task to be completed in advance of the Day of Move and the DOM Plan. This DOM Plan included hour by hour tasks, owners and prerequisites. The detailed Day of Move (DOM) Plan identified three move groups to reduce down time and limit risk. The Risk Management strategy included utilization of the disaster recovery center to maintain the required 24×7 Core Business Operations during the actual Move Event. In addition, the new location was integrated into the overall network infrastructure (via a new network infrastructure installed at the new location) so that all network capabilities were able to be tested in advance of the move. As much equipment as possible was installed in advance, and all cabling was prepositioned to reduce actual downtime on the Day of Move.
At every step along the way, this combined team of Client, MIGRA, and other Client vendor/partners worked as a coordinated unit to ensure that the Client Data Center and Headquarters Facility was ready when required.
Methodology
Pre-Move:
In the weeks prior to the move event, MIGRA conducted several site walk-throughs, cross checking the inventory sheet with the physical location of each device. The DOM Plan was reviewed during these site visits in conjunction with weekly scheduled Conf Calls where the team could ensure copies of all documents have been delivered and all task have been delivered on time. MIGRA determined the path for moving the devices and ordered sufficient floor covering material to protect the floor to and from the data center floor. i.e. carpet or tile. All packing materials were delivered to the site prior or on move day based on the DOM Plan requirements.
Move Day – Originating Location:
MIGRA personnel arrived at the site 2 hours early for a team meeting and site walk through. Floor covering was installed at the originating and destination locations. Two person teams were formed and assigned to areas/racks in the data center. A two person packing and moving team was responsible for receiving devices, cross checking inventory, packing and loading devices onto truck.
The MIGRA Logistics Manager coordinated the move day assignments with the on-site OEM Vendors who were responsible for the de-installation and re-installation and packing of some devices. MIGRA personnel were available to assist Vendors with crating the larger devices as needed.
As the Client released blocks of devices to be de-installed, technicians removed devices, cables, power cords and rails and moved them to the packing area. Devices were inventoried in and double checked to be sure the correct number of cables, power cords and rails were included. The devices with cords, cables and rails were packed and labeled with a Device Spec Sheet that included the name of the server, serial number, number of cables, cords, and rails, specific rack and U location at the destination.
To facilitate the delivery and re-installation of devices at the destination, the devices were placed on carts by destination rack, labeled and loaded into a truck. Although three trips had been initially planned, the execution was so efficient that only two move trips were required. The Early Group consisting of critical devices was loaded as soon as the devices were uninstalled and packed and dispatched to the destination site. Devices were unloaded at the destination and moved into place for re-installation. As soon as the first truck left the old building, the second truck was moved into position and the Core Environment was prepared for loading and transport. When it became clear that the second load would be the final load, the Project Manager directed the first team to remain at the new data center and continue with installation activities.
At the originating site, two technical teams continued to de-install and pack devices as they were released by the Client. Five racks were prepared for relocation and loaded onto the truck. The on-site vendor de-installed the OEM devices and turned them over to MIGRA for preparation and loading on the truck. When all of the devices and racks were de-installed, packed and loaded onto the truck, the truck was dispatched to the destination location.
Throughout the process at both the originating and destination locations, the devices were tracked by inventory number, noting time of de-installation, relocation and re-installation.
At the destination, the MIGRA installation teams were formed and assigned to work areas, aisle or racks. By working in teams on assigned racks, the technical teams were able to quickly install and cable the devices. A second team double checked the cabling and then turned the devices over to the Client team for testing.
High Priority devices were installed by the deadline, and the teams moved on to the other devices by priority group, completing the physical move within 8 hours – well under the estimated 12 hours.
After the equipment was racked in its proper places, the Client team and their other partners completed the process of connecting, testing and turning up the environment. By Sunday afternoon, Client was back into production in their new facility.
Lessons Learned
What Worked – Extensive Planning and Preparation:
The intensive planning and preparation period (6 months) took advantage of the three step process:
- Document the Requirements (PMO and DFS)
- Document the Design Plan
- Detailed Implementation Planning (Including 2 months of Day of Move Plan adjustments)
Prior to the Day of Move the Client conducted a successful test fail over of the core environment to the disaster recovery site. Network, voice and server test plans were executed. The destination environment was fully prepared with new racks, Ethernet and Fiber installed, hung and/or labeled. All net new servers were installed in the racks prior to the relocation of the existing devices.
A Device Spec Sheet was attached to each device with the rack number and location and with the number of cables to be installed. A sheet was posted on each rack listing the device name, Serial #, current rack #, current U location, all cable connections, the devices new rack #, U location, and all cable connections required.
The Day of Move Plan was followed line by line and quickly adjusted when we determined that we were well ahead of schedule.
The move was completed well under the estimated time giving the Client team more than sufficient time to execute all tests, complete replication and restore the operating environment.
What Could Have Been Improved – New Equipment Coordination:
The single area where we had an unexpected issue was that Client had installed one new rack and used some power that had been allocated to the rack next to the new one to power up a net new blade server enclosure. This enclosure had been used to facilitate testing. When we were installing the production blade enclosures in the production rack next to this new one, we came up short on power. Client quickly determined that they no longer needed the new test system and the power was restored to the production servers. Later, Client had their electrical contractor install additional power to that new rack.
Conclusion
In this engagement, every detail of pre-planning and execution was carefully planned and tested. The client dedicated appropriate resources to insure the planning was comprehensive. In some cases a client is unable to dedicate those resources. When that is the case, MIGRA is able to provide supplemental support to complete the planning effort. The client also assigned a professional project manager to this move. MIGRA’s project coordinator worked closely with the Client’s Project Manager to make sure every step of the project plan was completed in the defined order and milestones were met. In some cases a client may not have project management resources available to lead the effort. When that happens, the client’s management team is actively engaged as requested from start to finish. MIGRA was able to take their requirements and convert them into a customized Migration Plan designed to meet their deadlines. Project Management, Planning, Logistics, Resourcing and Leadership – the path to a successful Data Center Relocation is MIGRA!
