Creating a reliable and fast SAN network design with iSCSI

ISCSI is a versatile and affordable SAN access technology, but deployment encompasses more than just installing an iSCSI filer and connecting network nodes to it.

It helps to understand the Internet Small Computer System Interface (iSCSI) protocol before designing an iSCSI  network. In iSCSI, the blocks to be stored are sent over the network in an IP packet, rather than over dedicated SCSI cables. This iSCSI traffic is addressed on TCP port 3260 by default. The iSCSI block devices are configured on a node referred to as the iSCSI target, which is typically the storage-area network (SAN) in your environment. The servers connected to the SAN are iSCSI initiators.

An efficient SAN network needs sufficient bandwidth and speed, which means a dedicated iSCSI network. While iSCSI packets theoretically could travel on the user network, iSCSI's bursty nature requires a dedicated path so that packets are not lost. You should use mixed networks only in test environments.

Figure 1. This overview of the iSCSI network shows a redundant topology.

Consider the size of packets that will travel on the iSCSI network for your SAN. By default, an Ethernet packet is 1500 bytes, which really fragments SAN traffic. To optimize iSCSI traffic, configure all devices in the iSCSI network to use a maximum transmit unit of 9000 bytes -- the largest allowable at the Ethernet level. On some devices, this means enabling jumbo frames. Because the network requires 9000-byte packets throughout the blocks' journey, be sure to set this larger packet size at both ends of the communications channel. If one node is set to 1500-byte packets, you'll keep this smaller size throughout.

For the iSCSI network topology, use redundant networks and connect to redundant controllers if possible in your deployment. A redundant topology requires two network interfaces for each server that connect to the iSCSI network, with each interface connected to a separate switch. Two controllers must be available on the iSCSI SAN and connect each switch to each controller (see figure 1). To avoid having the same device presented several times via its redundant paths, install a multipath driver on the servers using the iSCSI network. With this redundancy, the nodes should survive a failure in the network topology.

After properly designing the iSCSI network for your SAN, consider optimization parameters available to your iSCSI target and the initiator. These will vary. For example, a Linux server connected to the iSCSI SAN has a configuration file for each active iSCSI connection in the /var/lib/iscsi directory, where you'll find multiple optimization parameters. An iSCSI host bus adapter that has an iSCSI offload engine will smooth out network traffic efficiently, preventing server latency while handling a high volume of traffic.

A well-designed iSCSI network won't need a great deal of iSCSI initiator optimization.

Sander van Vugt is an independent trainer and consultant based in the Netherlands. He is an expert in Linux high availability, virtualization and performance. He has authored many books on Linux topics, including Beginning the Linux Command Line, Beginning Ubuntu LTS Server Administration and Pro Ubuntu Server Administration.