The following Fibre Channel over Ethernet (FCoE) implementation case studies examine projects at three companies with a close eye on potential turf wars between the storage and networking staffs. These projects represent a portion of the full potential of the technology. So far, all three firms have implemented FCoE only between the servers and their FCoE switches, which split the LAN and the SAN data traffic. The SAN traffic goes through Fibre Channel (FC) core switches to FC-based storage. Nevertheless, the management challenges were big. Read these Fibre Channel over Ethernet implementation case studies and discover the issues these companies confronted and how they addressed them.
Table of contents
University of Arizona: Virtual networking team aids data center convergence
Kelsey-Seybold Clinic: Change to IBM one-stop IT shop provides challenges
Major media company: $1 million FCoE savings projection
University of Arizona: Virtual networking team aids data center convergence
The University of Arizona illustrates one of the smoother types of transitions to Fibre Channel over Ethernet (FCoE). The university used Ethernet and FC switches from Cisco Systems Inc. before and after implementing 10 Gigabit Ethernet (10 GbE)/FCoE. Although switching from Cisco Catalyst to Cisco Nexus switches represented change, that sort of shift is less likely to ruffle feathers than a vendor swap.
Background: Cost drove the university to shift to 10 GbE/FCoE between its servers and switches as part of an enterprise system replacement project. The Tucson-based school's SAN team drew up a convincing argument for FCoE. It estimated the cost to upgrade the existing FC infrastructure at $1.2 million, but a move to 10 GbE/FCoE at $600,000, largely due to a reduction in cables, adapters, switch ports and power consumption. The university has used Fibre Channel over Ethernet in production since July 2009.
Key technology: The IT team went with converged network adapters (CNAs) from QLogic Corp. and Cisco Nexus 5010 top-of-rack switches, Nexus 7000 core switches to aggregate Ethernet traffic and MDS
core switches for FC. The Nexus 5010s separate the LAN and SAN traffic. The university uses EMC Corp. storage.
Main management challenge: The LAN/WAN team and the SAN team were separate, reporting to different assistant directors. The teams would need to interact on a deeper level to succeed with FCoE, reasoned Derek Masseth, the university's senior director of frontline and infrastructure services.
"Our LAN/WAN team was a heavy-duty Cisco shop, much like most out there in the world, and they suffered from this preconception that if something said Cisco on it, they should manage it as a whole and they needn't talk to anyone else about the management thereof," Masseth said.
"That flew in the face of the approach that the SAN team felt they needed to take," he continued. The SAN team's thought process was like this: 'If the delivery of our service, which is fundamentally storage over Fibre Channel, is going to be dependent on this stuff, we need to not only be aware of what's going on but we need to be intimately engaged in its configuration.' They came to the table demanding that their silo be preserved and they get what they need in the management of the physical layout of the SAN and of the logical configuration."
How IT addressed the challenge: With data center convergence in mind, the university orchestrated a departmental shift at least a year before the Fibre Channel over Ethernet procurement. Both the LAN/WAN and SAN teams report directly to Masseth and share a common ticketing system.
Masseth carved out a data center networking "virtual team" spanning the two groups and made a concerted effort to introduce them and facilitate relationship building, including "touchy feely" managerial activities such as lunch with the heads of the two teams.
"This technology is transformational in its very nature, and it demands that we transform our organization in response," Masseth recalled telling them. "It was one of the most difficult components of the transition, really convincing the folks on the ground that we needed to fundamentally change the way we think about each other and how we interact."
The managers gave technical staffers the time and resources to get up to speed on the new technology. They held meetings. They brought in a consultant. They sent four key staffers to off-site five-day training seminars in pairs (one from the LAN team and one from the SAN team) so they could get to know one another.
Management encouraged Ken Johnson, a senior communications network analyst on the LAN team, and his SAN counterpart, Adam Michel, to draw up the division of labor for managing the new FCoE infrastructure. Johnson came up with a template for new switch configurations, Michel took responsibility for the virtual SAN, and they work together to verify proper FCoE operation.
Long-term management plan: Masseth doesn't expect the LAN/WAN and SAN to ever consolidate to a single team. But he does foresee the virtual team becoming a real entity on the organizational chart.
Masseth said he ultimately envisions the university going FCoE all the way to the storage, and, at that point, the SAN team might win out "because they understand the issues that are most directly encountered inside of the data center" and the LAN team could return to their traditional responsibilities.
Kelsey-Seybold Clinic: Change to IBM one-stop IT shop provides challenges
The Houston-based Kelsey-Seybold Clinic learned that a shift to Fibre Channel over Ethernet presents additional management challenges when paired with a change to a network technology vendor different from the one with which staff members have special expertise and experience.
Background: Kelsey-Seybold Clinic, a healthcare provider with 19 locations in the greater Houston area, decided to implement a 10 GbE network and FCoE in connection with a move to a new data center located approximately 100 miles away in Bryan, Texas, away from the hurricane zone.
Key technology: The IT team sought to become a one-stop shop and chose IBM to supply its servers, CNAs, rebranded Brocade switches and storage, as well as support services for those products. Some staffers viewed IP switches as a commodity and gave Brocade a slight edge because of its FC heritage and the maturity of its DCX Backbone/IBM SAN768B. The clinic went live with FCoE with some applications in June 2009 and expects to complete the migration within 12 months..
"From a financial perspective and from a bigger-picture perspective, we felt we were making the decision for the greater good [putting] everything underneath the IBM umbrella," said Tony Taylor, manager of enterprise network services on the LAN/WAN side at Kelsey-Seybold Clinic. Sticking with one vendor also gave them a better shot at using common tools for managing and monitoring.
Main management challenge: The IT department needs to work out roles and responsibilities for managing the converged network. The LAN/WAN network team has extensive experience and certifications working with Cisco's Ethernet switches, so the change to Brocade was a bitter pill for some to swallow.
How IT addressed the challenges: Kelsey-Seybold Clinic's organizational structure was conducive to FCoE, since the LAN/WAN team and the SAN/storage/server administration team report to the same manager, Martin Littmann, the director of IT systems. Littmann enlisted the managers of both teams and as many members of the technical staff as possible in the decision-making process and encouraged them to ask tough questions of vendors.
The storage team, which also provisions servers, and the LAN/WAN team have approximately 20 people between them, so they've always tended to work in close quarters and collaborate. The managers of the two teams literally sit next to each other, and their teams meet together.
"My expectation is we function as one single team," Littmann said.
Keeping the lines of communication open and promoting discussion have been critical to soothe any lingering hard feelings over the shift to the IBM-rebranded Brocade switches. Both teams received external and internal training.
"While it certainly hasn't been without challenges, both teams at this point are seeing this as an opportunity to seriously cross-train and gain extra insight into areas that they have not traditionally been a part of," Taylor said.
Kelsey-Seybold Clinic currently leans more heavily on the storage team to manage the fabric while the networking team gets up to speed. But Littmann said that could transition over time.
Long-term management plan: Littmann said he expects the LAN and SAN teams to continue to share responsibility for managing FCoE. Kelsey-Seybold Clinic also has asked product manufacturers to work on additional role-based administration features to assist with management.
Major media company: $1 million FCoE savings projection
Trying to implement FCoE on an especially aggressive schedule might prompt a need for outside help. The project lead at a major media company decided to bring in consultants with special expertise in Cisco technology to get the work done more quickly. However, the firm trained key in-house staff for the ongoing operation, easing a potentially prickly transition by letting the storage and network teams continue to manage their parts of their infrastructure.
Background: The media company replaced its aging FC infrastructure with next-generation Fibre Channel over Ethernet in connection with a move to a new data center. Anthony Kwan, the firm's executive IT advisor and solution architect for network and security, judged that FCoE would provide flexibility, cost savings and bandwidth to facilitate future growth.
Key technology: The media company is one the first Cisco customers in its geographic region to use the Cisco Data Center 3.0 framework, with its Cisco Unified Computing System (UCS) for servers, CNAs, fabric extenders and management software, as well as Nexus 5020 top-of-rack switches and Nexus 7010 core switches. The IT team uses Brocade 12000 switches to connect to its NetApp storage via FC, but plans to retire the 12000s in favor of Cisco's MDS core switches when the latter support FCoE, according to Kwan.
The company deployed the Cisco Unified Computing System in production last fall and the Nexus 5020s and Nexus 7010s in May. It intends to use the Nexus 7000 for native FCoE back and forth with the MDS switches in the coming months, shortly after the technology is ready, Kwan said. The company ultimately plans to extend FCoE to its storage once the technology is ready, he added.
Main management challenge: The need to do more with less took on heightened importance in an industry beset by financial difficulties even before the economy soured. Kwan first had to convince upper management that FCoE and the Cisco Unified Computing System and switches would bring significant savings and benefits, then figure out the best way to implement and maintain the new technology with an IT shop that had undergone significant staff reductions.
How IT addressed the challenge: Kwan joined the media company last year with a mission to identify and analyze the data center requirements, develop the new data center architecture and recommend technology. He made a case that FCoE would bring more than $1 million in savings through a reduction in FC cables and adapters. He estimated substantial additional savings through ease of management, scale and the added business capabilities the new architecture would afford.
The media company's IT department was already accustomed to radical change after undergoing a 50% staff reduction the prior year. The FCoE/Cisco UCS step brought an additional 17% reduction in the network IT staff, according to Kwan.
"We have less people doing more work," Kwan said, noting that a smaller staff now brings up more virtual machines in a shorter timeframe. Kwan, who is a Cisco Certified Internetwork Expert (CCIE) in three technology areas, drew up a detailed implementation plan and hired two CCIE consultants to help with the implementation, after determining in-house IT staff didn't have the skill set necessary to do the work in the projected four-and-a-half-month timeframe. The trio completed the work in six weeks, according to Kwan.
Meanwhile, the in-house staff received training to operate the new technology. Hand-picked engineers went to class and then shared the knowledge with colleagues upon completion. Kwan said he carved out a transition period in his implementation plan and conducted more than 20 information architecture sharing sessions.
Long-term management plan: Kwan said he sees no reason to change the management structure, so long as the network and storage staffers understand the technology. The two teams remain separate and report to the same managers as they did before Fibre Channel over Ethernet.
"There is a reason why they work in their own technology verticals. Networking knowledge and storage expertise are still diverse," Kwan said. He added that separate IT teams have always had to troubleshoot problems over lengthy conference calls, so he doesn't see how the situation will be any different with FCoE.
Dig Deeper on Primary storage devices
