Linux DM Multipath
Multipathing (DM-MPIO) provides input-output (I/O) fail-over and load-balancing within Linux for block devices. By utilizing device-mapper, multipathd provides the host-side logic to use multiple paths of a redundant network to provide continuous availability and higher bandwidth connectivity between the host server and the block-level device. DM-MPIO handles the rerouting of block I/O to an alternate path in the event of a path failure. DM-MPIO can also balance the I/O load across all of the available paths that are typically utilized in Fibre Channel (FC) and iSCSI SAN environments.

DM-MPIO in Linux consists of kernel components and user-space components.
Kernel – device-mapper – block subsystem that provides layering mechanism for block devices.

User-space – multipath-tools – provides the tools to manage multipathed devices by instructing the device-mapper multipath module what to do. The tools consist of:
Multipath: scans the system for multipathed devices, assembles them, updates the device-mapper’s map. . And Lists and configures multipath devices. Normally started up with /etc/rc.sysinit, it can also be started up by a udev program whenever a block device is added
Multipathd: daemon that waits for maps events, and then executes multipath and monitors the paths. Marks a path as failed when the path becomes faulty. Depending on the failback policy, it can reactivate the path. And also Monitors paths; as paths fail and come back, it may initiate path group switches. Provides for interactive changes to multipath devices. This must be restarted for any changes to the /etc/multipath.conf file.
Devmap-name: provides a meaningful device-name to udev for devmaps.
Kpartx: maps linear devmaps to device partitions to make multipath maps partionable. Creates device mapper devices for the partitions on a device It is necessary to use this command for DOS-based partitions with DM-MP. The kpartx is provided in its own package, but the device-mapper-multipath package depends on it
Multipath.conf: configuration file for the multipath daemon. Used to overwrite the built-in configuration table of multipathd.
mpathconf utility Configures and enables device mapper multipathing.

Configuration file
The configuration file /etc/multipath.conf makes many of the DM-MPIO features user-configurable. The multipath command and the kernel daemon multipathd use information found in this file. The file is only consulted during the configuration of the multipath devices. Changes must be made prior to running the multipath command. Changes to the file afterwards will require multipath to be executed again.
The multipath.conf has five sections:
1. System level defaults (defaults): User can override system level defaults.
2. Blacklisted devices (blacklist): User specifies the list of devices that is not be to under the control of DM-MPIO.
3. Blacklist exceptions (blacklist_exceptions): Specific devices to be treated as multipath devices even if listed in the blacklist.
4. Storage controller specific settings (devices): User specified configuration settings will be applied to devices with specified “Vendor” and “Product” information.
5. Device specific settings (multipaths): Fine tune the configuration settings for individual LUNs.

Multipath Device Identifiers
Each multipath device has a World Wide Identifier (WWID), which is guaranteed to be globally unique and unchanging. By default, the name of a multipath device is set to its WWID. Alternately, you can set the user_friendly_names option in the multipath configuration file, which sets the alias to a node-unique name of the form mpathn.

For example, a node with two HBAs attached to a storage controller with two ports via a single unzoned FC switch sees four devices: /dev/sda, /dev/sdb, dev/sdc, and /dev/sdd. DM-Multipath creates a single device with a unique WWID that reroutes I/O to those four underlying devices according to the multipath configuration. When the user_friendly_names configuration option is set to yes, the name of the multipath device is set to mpathn.

When new devices are brought under the control of DM-Multipath, the new devices may be seen in two different places under the /dev directory: /dev/mapper/mpathn and /dev/dm-n.

• The devices in /dev/mapper are created early in the boot process. Use these devices to access the multipathed devices, for example when creating logical volumes.

• Any devices of the form /dev/dm-n are for internal use only and should never be used.

You can also set the name of a multipath device to a name of your choosing by using the alias option in the multipaths section of the multipath configuration file. For information on the multipaths section of the multipath configuration file,

The basic procedure for configuring your system with DM-Multipath is as follows:
• Install the device-mapper-multipath rpm.
• Create the configuration file and enable multipathing with the mpathconf command. You can also start the multipath daemon with this command if you do not need to edit the configuration file.
• If necessary, edit the multipath.conf configuration file to modify default values and save the updated file.
• Start the multipath daemon.
To Enable Multipathing in Linux
Use the Linux command modprobe to add the loadable kernel modules dm-multipath and dm-round-robin.
# modprobe dm-multipath
# modprobe dm-round-robin
# service multipathd start
List the multipath devices that have been created.
# multipath -ll
The output should list the same number of devices as there are disks in the J4500 array. The following is an example of output:
35000c5000357625b dm-2 SEAGATE,ST340008SSUN0.4
[size=373G][features=0][hwhandler=0]
\_ round-robin 0 [prio=2][active]
\_ 0:0:0:0 sda 8:0 [active][ready]
\_ 1:0:0:0 sdm 8:192 [active][ready]

To Disable Multipathing in Linux
1. If a RAID volume, LVM volume, or volume mount have been placed over the device node of the multipathed disk, quiesce the volume.
2. Use the multipath -f command to disable multipathing to a specific device.
# multipath -f mpath1
3. Use the multipath -F command to disable multipathing on all multipathed devices.
# multipath -F

Additional Information( Trouble Shooting)
Determining Device Mapper Entries with the dmsetup Command

The following command displays all the device mapper devices and their major and minor numbers. The minor numbers determine the name of the dm device. For example, a minor number of 3 corresponds to the multipathed device /dev/dm-3.

# dmsetup ls
mpathd (253:4)
mpathep1 (253:12)
mpathfp1 (253:11)
mpathb (253:3)
mpathgp1 (253:14)
mpathhp1 (253:13)
mpatha (253:2)
mpathh (253:9)
mpathg (253:8)
VolGroup00-LogVol01 (253:1)
mpathf (253:7)
VolGroup00-LogVol00 (253:0)
mpathe (253:6)
mpathbp1 (253:10)
mpathd (253:5)
Troubleshooting with the multipathd Interactive Console
The multipathd -k command is an interactive interface to the multipathd daemon. Executing this command brings up an interactive multipath console. After executing this command, you can enter help to get a list of available commands, you can enter an interactive command, or you can enter CTRL-D to quit.
The multipathd interactive console can be used to troubleshoot problems you may be having with your system. For example, the following command sequence displays the multipath configuration, including the defaults, before exiting the console.
# multipathd -k
> > show config
> > CTRL-D
The following command sequence ensures that multipath has picked up any changes to the multipath.conf,

# multipathd -k
> > reconfigure
> > CTRL-D
Use the following command sequence to ensure that the path checker is working properly.
# # multipathd -k
> > show paths
> > CTRL-D

Table 5.1. Useful multipath Command Options
Option Description
-l Display the current multipath configuration gathered from sysfs and the device mapper.
-ll Display the current multipath configuration gathered from sysfs, the device mapper, and all other available components on the system.
-f device Remove the named multipath device.
-F Remove all unused multipath devices.
-w device (Red Hat Enterprise Linux 6.6 and later) Remove the wwid of the specified device from the wwids file.
-W (Red Hat Enterprise Linux 6.6 and later) Reset the wwids file to include only the current multipath devices.

# lsscsi
[1:0:0:0] disk SEAGATE ST330055SSUN300G 0B92 /dev/sda
[1:0:1:0] disk SEAGATE ST330055SSUN300G 0B92 /dev/sdb
[1:0:2:0] disk SEAGATE ST330055SSUN300G 0B92 /dev/sdc
[1:0:3:0] enclosu SUN Storage J4500 3R21 –
[2:0:0:0] disk SEAGATE ST330055SSUN300G 0B92 /dev/sdd
[2:0:1:0] disk SEAGATE ST330055SSUN300G 0B92 /dev/sde
[2:0:2:0] disk SEAGATE ST330055SSUN300G 0B92 /dev/sdf
Once the multipath daemon is started on the host, you can see multipath details using the multipath command.
# multipath -ll
35000c5000357625b dm-2 SEAGATE,ST340008SSUN0.4
[size=373G][features=0][hwhandler=0]
\_ round-robin 0 [prio=2][active]
\_ 1:0:1:0 sdb 8:0 [active][ready]
\_ 2:0:1:0 sde 8:192 [active][ready]

# lsscsi -g
[1:0:0:0] disk SEAGATE ST330055SSUN300G 0B92 /dev/sda /dev/sg0
[1:0:1:0] disk SEAGATE ST330055SSUN300G 0B92 /dev/sdb /dev/sg1
[1:0:2:0] disk SEAGATE ST330055SSUN300G 0B92 /dev/sdc /dev/sg2
[1:0:3:0] enclosu SUN Storage J4500 3R21 – /dev/sg3
[2:0:0:0] disk SEAGATE ST330055SSUN300G 0B92 /dev/sdd /dev/sg4
[2:0:1:0] disk SEAGATE ST330055SSUN300G 0B92 /dev/sde /dev/sg5
[2:0:2:0] disk SEAGATE ST330055SSUN300G 0B92 /dev/sdf /dev/sg6