Case Study: Managing the Logistics
Summary
A national entertainment and hospitality corporation planned to migrate a regional datacenter to the headquarters facility. The Client assigned an experienced IT Project Manager and Logistics Manager to lead an internal project team. The Client team developed a comprehensive migration and implementation plan, identifying application dependencies, rack elevations, cabling plans and migration schedule. Once the plan was complete, the Client contacted MIGRA to handle the logistics.
Project Requirements
The requirement: Relocate 400 servers and other devices over 1,600 miles in a 24 hour window. The Client had a top notch IT team who worked for months on the plan to migrate data and hardware from the regional center to headquarters. They had identified their requirements, prepped the site, dropped cables and labeled locations in the racks for hardware installation. The Client’s IT PM and Logistics PM developed a detailed plan that would meet the business requirements of minimum downtime, now they needed a company to actually move the devices. When considering firms to do the work, they realized they wanted more than a transportation company; they wanted a company that could join their team. The company would be able to quickly review their logistics requirements, provide alternative solutions with the pros and cons both for time and cost, and then follow through and meet the deadlines. MIGRA was the right match. Our team of professionals was able to review their logistics plan, provide a detailed range of options including cost alternatives and levels of risk for each choice. MIGRA understands the work that was done behind the final logistics plan and could ask good questions about the technical and physical details that had to be considered to meet the deadlines.
A test move had gathered detailed data on time required for de-installation, packing, relocation, unpacking and re-installation. The data was used to determine the number of technicians, movers and trucks required to complete the move in the allocated time. The move was accomplished by creating teams of technicians and movers at the originating and destination locations. A chartered DC-9 cargo plane was used to fly the devices to the destination. The core team, including the three owners, flew by chartered plane at the same time to meet the cargo plane and supervise the unloading and transfer to the destination. By having the owners on-site working with the technical and moving teams, MIGRA could quickly resolve any issues that arose during the move. The MIGRA team coordinated all of the personnel, site preparation, transportation and cargo handling logistics for the move. The move was accomplished on time with all of the devices installed or stored ahead of schedule allowing the Client’s technicians and vendors’ additional time for testing before bringing the systems back online.
About the Move
The devices to be relocated included EMC storage arrays, blades, optical storage devices, IBM and Wintel servers and related devices such as PDUs, KVMs and network devices. The requirements for the move included a 24 hour move window for 30% of the devices and 48 hours for the remainder. This was a high-value, high-security move requiring a team of qualified technicians and move personnel and the secure, exclusive use of all transportation modes (plane or truck).
The Client Team was responsible for the project planning including disaster recovery strategies. The devices to be relocated would be integrated into the existing data center design and rack layout at the Headquarters Data Center. The Client provided a detailed listing by move group of all devices, locations, and network connections.
One month prior to the primary move, MIGRA conducted a test move of 40 devices. MIGRA documented each move step to develop a per device time estimate to be used in determining staffing for the primary move. The Client clustered the devices into the three move groups distributed throughout the data center closely representing the conditions for the final move. The test move was accomplished within the specified move window with all devices re-installed and operational in less than 24 hours. After reviewing the time estimates from the Test Move, MIGRA implemented several time saving options to reduce time required for the overall move. To meet the required deadlines, the final logistics plan had three move groups – high priority, medium priority, and priority. Within the high priority group there were first, second and third groups listed in priority order for re-installation.
Methodology
Pre Move Day:
MIGRA conducted a site walk through cross checking the inventory sheet with the physical location of each device. The Move Day plan was reviewed with the Client to ensure copies of all documents would be on hand on Move Day. MIGRA determined the path for moving devices and ordered sufficient floor covering material to protect the floor of the data center from scratching. All packing materials were previously delivered to the site.
Move Day – Originating Location:
MIGRA personnel arrived at the site 2 hours early for a team meeting and site walk through. Two person teams were formed and assigned to areas of the data center. Two person packing and moving teams would be responsible for receiving devices, cross checking inventory, packing and loading devices.
MIGRA coordinated the move day assignments with on-site Vendors who would be responsible for the de-installation and/or de-installation and packing of OEM devices. MIGRA personnel were available to assist Vendors with crating the larger devices.
Teams prepped the packing boxes based on the inventory device count. Boxes and packing material were staged in a central area.
As the Client released blocks of devices to be de-installed, technicians removed devices, cables, power cords and rails and moved them to the packing area. Devices were inventoried in and double checked to be sure the correct number of cables, power cords and rails were included. The devices with cords, cables and rails were packed and labeled on the outside of the box with a Device Spec Sheet that included the name of the server, serial number, number of cables, cords, rails and specific rack and U location at the destination.
To facilitate the delivery and reinstallation of devices at the destination, the devices were palletized by move category, labeled and loaded into a truck. For this move, two trucks were used, sending the first truck to the air cargo facility to begin loading the pallets into the cargo plane. When all devices were de-installed and loaded onto the truck, the second truck was sent to the cargo facility to begin loading.
Move Day – Destination Location:
MIGRA personnel accompanied the flight, arriving to supervise unloading the plane and transferring to ground transportation. Priority devices were loaded on to the first truck and dispatched to the Data Center. The remainder of the devices were loaded on to the second truck and dispatched to the Data Center. The trucks were unloaded into the staging areas and move teams began unpacking the highest priority devices, inventorying them in, testing to the OS and scheduling them to go into the data center for installation.
At the Data Center, the MIGRA installation teams were briefed on the re-installation process and rack locations. Two person technical teams were formed and assigned to work areas. By working in teams in assigned areas of the data center, technical teams were able to quickly install and cable the devices. A second team double checked the cabling and then turned the devices over to the Client team for testing.
High Priority devices were installed by the deadline, and the teams moved on to the other devices by priority group, completing the move within 30 hours well under the estimated 48 hours.
Lessons Learned
What Worked – Extensive Planning and Preparation:
The Client had completed an intensive planning process that had followed the three step process:
- Document the Present Method of Operation;
- Define the Desired Future State;
- Implementation Plan.
A test move of a number of devices provided needed input on man hours required for de-installation and re-installation. The Implementation Plan timeline was adjusted to more accurately reflect actual time required.
Streamlining Move Day
The Client provided detailed inventory lists with the name of the device, serial number, u-size, rack number, rack location, and the number and type of cables. On the day of move, MIGRA color coded the inventory sheets and move scripts for the devices. In lessons learned from the test move, the devices were numbered (1, 2, 3,…). In the test move, it took additional time to verify the correct device to remove by serial number and then checking the device against inventory before packing. Using the numbering system it was much quicker to find the device on the inventory list and do a quick verification that it matched the asset number on the inventory sheet.
Each device was packed in a separate box with rail kits, screws and any cables. A Device Spec Sheet was inserted in a packing envelop and affixed to the sealed box. The installation priority number was written in marker on the side and top of the box. Boxes were placed on the pallets, keeping the installation priority groups together. After wrapping the pallets, a second label was placed on the pallet, color coded to match the re-installation move group. When the plane was unloaded high priority pallets would be loaded on the first transport truck and sent to the facility.
Boxes were prepared in advance of the Move Start time and sorted by U-size, bubble wrap was cut into sheets and move scripts were placed into packing envelopes and sorted by device number and move group.
At the originating location, the technicians and movers arrived the day before the priority move for orientation training and to de-install devices to be decommissioned or that were not in production. This provided hands on training on MIGRA’s methodology and requirements for de-installation and packing.
The de-installation was scheduled to take 12 hours from start to finish, a long day for the technicians and movers. Because the teams were more efficient than planned, a one hour down time occurred around mid-point giving everyone a break.
MIGRA reserved the DC-9 cargo plane for an earlier start time than scheduled to ensure that the plane would be on the ground on time allowing for weather and air traffic. This allowed the first truck with the lower priority devices to begin loading cargo earlier than scheduled. The first phase of the move was completed almost 1 hour ahead of schedule adding a window of time for any loading or flight departure delays.
At the receiving location, MIGRA reserved the cargo handlers and transport trucks one hour before scheduled arrival to ensure readiness to unload if there was an early arrival. As it turned out, the cargo plane arrived before the 6am estimated time, allowing a quick unloading and departure from the cargo terminal.
Fresh teams of technicians along with the core MIGRA team leads and Technicians and our local truck agent unloaded and re-installed devices at the final destination. The Device Spec Sheet included the rack number and U location, with the number of cables to be installed. The Client posted a sheet on each rack listing the name and location for all devices to be installed.
What Could have been Improved:
Checking off devices on the inventory sheet could have been improved by posting the inventory sheets on a board to keep them from getting separated or misfiled. This would also be a quick visual check for the Client to be sure both inventories were in sync.
Attaching the correct move script to the device could have been improved by pre-inserting the scripts into the packing envelopes before filing the scripts in order.
At the destination, the staging area was physically separated from the cage and cell service was limited. The move crew was able to unpack, test and queue the devices on the moving carts quickly. However, there was an insufficient work area in the cage to safely unload and stage the devices. This led to a less than optimum work area for the technicians. Hand held communicators would have helped solve this problem allowing the lead technician in an aisle to call for the next group at the appropriate time.
The hot aisles were narrow and poorly lit and quickly became crowded when two or more technicians were working in the area. If teams of installers remained in the cold aisles installing racks and devices, a separate team could be tasked to cable devices when installed in the hot aisle.
Conclusion
This move demonstrated the major components that are required for a successful Data-Center-Relocation/Migration. These include project management, solid planning resources, detailed logistics coordination, management support and appropriate resources. In this engagement every detail of pre-planning and execution was carefully planned and tested. This client dedicated appropriate resources to insure the planning was comprehensive. In some cases a client is unable to dedicate those resources. When that is the case, MIGRA, in conjunction with the client is able to provide supplemental support to complete the planning effort. The client also assigned a professional project manager to this move. MIGRA’s Project Manager worked closely with the Client Project Manager to make sure every step of the project plan was completed in the defined order and milestones were met. In some cases a client may not have project management resources available to lead the effort. When that happens, MIGRA is able to provide professional project management support as needed. The client’s management team was actively engaged in this project from start to finish which was how the appropriate resources were able to perform as well as they did. Finally, this client selected MIGRA as service provider due in part to our experience, resources and hands on approach. MIGRA was able to take their requirements and convert them into a customized Migration Plan designed to meet their deadlines. Project Management, Planning, Logistics, Resourcing and Leadership – the path to a successful Data Center Relocation is MIGRA, Inc!
