Mdadm: Difference between revisions

Browse history interactively ← Previous editContent deleted Content addedVisual WikitextInline

Revision as of 07:24, 13 June 2013 edit14.140.2.178 (talk) reducing localization← Previous edit		Latest revision as of 08:12, 11 June 2024 edit undoCitation bot (talk \| contribs)Bots5,445,456 edits Added website. \| Use this bot. Report bugs. \| Suggested by LeapTorchGear \| #UCB_webform 139/234
(139 intermediate revisions by 71 users not shown)
Line 1:		Line 1:
			{{Short description\|Linux utility for managing software RAID}}
	{{multiple issues\|
	{{~~more~~ footnotes\|date=November 2011}}		{{More footnotes\|date=November 2011}}
	{{~~Howto\|date=May~~ ~~2010~~}}		{{Lowercase title}}
	}}

	{{lowercase}}

	{{Infobox software		{{Infobox software
	\| name = mdadm		\| name = mdadm
	\| author = Neil Brown		\| author = Neil Brown
			\| developer = Community contributors, <br /> Mariusz Tkaczyk<ref>{{cite web \| url = https://marc.info/?l=linux-raid&m=170256349004550&w=2 \| title = Announcement: mdadm maintainer update \| date = 2023-12-14 \| accessdate = 2024-05-17 \| publisher = marc.info }}</ref>
	\| developer = Neil Brown
	\| released = 2001		\| released = 2001
			\| latest release version = 4.3<ref>{{cite web \| url = https://git.kernel.org/pub/scm/utils/mdadm/mdadm.git/commit/?id=d709d4161d1dd51a976147e8372fbd99ff8183bd \| title = Release mdadm-4.3 \| date = 2024-02-15}}</ref>
	\| latest release version = 3.2.6<ref>{{cite web
			\| latest release date = {{release date and age\|2024\|02\|15}}
	\| url=http://comments.gmane.org/gmane.linux.raid/40535
			\| repo = {{URL\|github.com/md-raid-utilities/mdadm/}}
	\| title=ANNOUNCE: mdadm 3.2.6 - A tool for managing Soft RAID under Linux
			\| programming language = ]
	\| date=2012-10-25
			\| operating system = ]
	\| first=Neil
			\| language = ]
	\| last=Brown}}</ref>
			\| genre = Disk utility
	\| latest release date = {{release date and age\|2012\|10\|25}}
			\| license = ]
	\| programming language = ]
	\| operating system = ]
	\| language = ]
	\| status = Active
	\| genre = Disk utility
	\| license = ]
	\| website = http://neil.brown.name/blog/mdadm
	}}		}}
	'''mdadm''' is a ] utility used to manage ] devices.

			'''mdadm''' is a ] utility used to manage and monitor ] devices. It is used in modern ]s in place of older software RAID utilities such as ] or ].<ref name="lpic-2">{{cite book\|url=https://books.google.com/books?id=YBktDQAAQBAJ&q=mdadm\|pages=206–221\|title=LPIC-2: Linux Professional Institute Certification Study Guide\|publisher=]\|first1=Christine\|last1=Bresnahan\|first2=Richard\|last2=Blum\|year=2016\|isbn=9781119150817}}</ref><ref>{{cite book\|url=https://archive.org/details/managingraidonli0000vada\|url-access=registration\|page=\|quote=mdadm linux.\|title=Managing RAID on Linux\|first=Derek\|last=Vadala\|publisher=O'Reilly Media, Inc\|year=2003\|isbn=9781565927308}}</ref><ref>{{cite book\|url=https://books.google.com/books?id=0SIdBAAAQBAJ&q=%22mdadm%22+%22linux%22&pg=PA244\|title=UNIX and Linux System Administration Handbook\|first=Evi\|last=Nemeth\|pages=242–245\|isbn=9780131480056\|publisher=Pearson Education\|year=2011}}</ref>
	The name is derived from the '''md''' (multiple device) device nodes it '''adm'''inisters or manages, and it replaced a previous utility ''mdctl''. The original name was "Mirror Disk", but was changed as the functionality increased.

			mdadm is ] originally<ref>{{Cite web\|url=http://neil.brown.name/blog/mdadm\|title=Mdadm\|access-date=2007-08-25\|archive-date=2013-05-03\|archive-url=https://web.archive.org/web/20130503183707/http://neil.brown.name/blog/mdadm\|url-status=dead}}</ref> maintained by, and copyrighted to, Neil Brown of ], and licensed under the terms of version 2 or later of the ].
	It is ] licensed under version 2 or later of the ] - maintained and copyrighted to Neil Brown of ].

			==Name==
	{{Portal\|Free software}}
			The name is derived from the ''md'' (multiple device) device nodes it ''administers'' or manages, and it replaced a previous utility ''mdctl''.{{citation needed\|date=December 2016}} The original name was "Mirror Disk", but was changed as more functions were added.{{citation needed\|date=December 2016}} The name is now understood to be short for Multiple Disk and Device Management.<ref name="lpic-2" />

	==~~Functionality~~==		== Overview ==
			Linux software RAID configurations can include anything presented to the ] as a ]. This includes whole hard drives (for example, {{mono\|/dev/sda}}), and their ] (for example, {{mono\|/dev/sda1}}).

	===~~Types~~ of ~~physical~~ ~~device~~===		=== RAID configurations ===

	mdadm can handle anything which presents to the ] as a ]. This can encompass whole disks (/dev/sda) and ] (/dev/sda1).

	===RAID Configurations===
	{{Main\|Standard RAID levels}}		{{Main\|Standard RAID levels}}
			{{See also\|Linux MD RAID 10}}

	* RAID 0 - Block level ]. MD can handle devices of different lengths, the extra space on the larger device is then not striped.		* ''RAID 0'' – Block-level ]. MD can handle devices of different lengths, the extra space on the larger device is then not striped.
	* RAID 1 - Mirror.		* ''RAID 1'' – Mirror.
	* RAID 4 - Like RAID 0, but with an extra device for ].		* ''RAID 4'' – Like RAID 0, but with an extra device for the ].
	* RAID 5 - Like RAID 4, but with the parity distributed across all devices.		* ''RAID 5'' – Like RAID 4, but with the parity distributed across all devices.
	* RAID 6 - Like RAID 5, but with two parity segments per stripe.		* ''RAID 6'' – Like RAID 5, but with two parity segments per stripe.
	* RAID 10 - Take a number of RAID 1 mirrorsets and stripe across them RAID 0 style.		* ''RAID 10'' – Take a number of RAID 1 mirrorsets and stripe across them RAID 0 style.

	~~(Note that "~~RAID 10" is distinct from RAID "0+1", which consists of a top-level RAID-1 mirror composed of high-performance RAID-0 stripes directly across the physical hard disks. A single-drive failure in a RAID 10 configuration results in one of the lower-level mirrors entering ], but the top-level stripe performing normally (except for the performance hit). A single-drive failure in a RAID 0+1 configuration results in one of the lower-level stripes ''completely failing'', and the top-level mirror entering degraded mode. Which of the two setups is preferable depends on the details of the application in question, such as whether or not spare disks are available, and how they should be spun up.)		''RAID 10'' is distinct from ''RAID 0+1,'' which consists of a top-level RAID 1 mirror composed of high-performance RAID 0 stripes directly across the physical hard disks. A single-drive failure in a RAID 10 configuration results in one of the lower-level mirrors entering ], but the top-level stripe performing normally (except for the performance hit). A single-drive failure in a RAID 0+1 configuration results in one of the lower-level stripes ''completely failing,'' and the top-level mirror entering degraded mode. Which of the two setups is preferable depends on the details of the application in question, such as whether or not spare disks are available, and how they should be spun up.

	===Non-RAID ~~Configurations~~===		=== Non-RAID configurations ===
			* ''Linear'' – concatenates a number of devices into a single large MD device – (deprecated since 2021 and removed from the Linux kernel since 2023 <ref name="linux-linear-removal">{{cite web \| url = https://github.com/torvalds/linux/commit/849d18e27be9a1253f2318cb4549cc857219d991 \| title = md: Remove deprecated CONFIG_MD_LINEAR \| website = ] \| date = 2023-12-19 \| accessdate = 2024-04-30 }}</ref>)
			* ''Multipath'' – provides multiple paths with failover to a single device
			* ''Faulty'' – a single device which emulates a number of disk-fault scenarios for testing and development
			* ''Container'' – a group of devices managed as a single device, in which one can build RAID systems

			== Features ==
	* LINEAR – Concatenate a number of devices into a single large MD device.
			The original (standard) form of names for md devices is {{mono\|/dev/md<n>}}, where {{mono\|<n>}} is a number between 0 and 99. More recent kernels have support for names such as {{mono\|/dev/md/Home}}. Under 2.4.x kernels and earlier these two were the only options. Both of them are non-partitionable.
	* MULTIPATH – Provide multiple paths with failover to a single device.
	* FAULTY – A single device which emulates a number of disk fault scenarios for testing and development.
	* CONTAINER – A group of devices managed as one, in which RAID systems can be built.

			Since 2.6.x kernels, a new type of MD device was introduced, a ] array. The device names were modified by changing {{mono\|md}} to {{mono\|md_d}}. The partitions were identified by adding {{mono\|p<n>}}, where {{mono\|<n>}} is the partition number; thus {{mono\|/dev/md/md_d2p3}} for example. Since version 2.6.28 of the ], non-partitionable arrays can be partitioned, the partitions being referred to in the same way as for partitionable arrays{{snd}} for example, {{mono\|/dev/md/md1p2}}.
	===Types of MD device===

			Since version 3.7 of the Linux kernel mainline, md supports ] operations for the underlying ]s (SSDs), for linear, RAID 0, RAID 1, RAID 5 and RAID 10 layouts.<ref>{{cite web
	The original (standard) form was '''/dev/md''n''''' where '''''n''''' is a number between 0 and 99. More recent kernels have supported the use of names such as '''/dev/md/Home'''. Under kernel 2.4 and earlier these two were the only options. Both of them are non-partitionable.
			\| url = http://kernelnewbies.org/Linux_3.7#head-2fd9b183a4623d96e69ed24f88e0eb83217fa8df
			\| title = Linux kernel 3.7, Section 5. Block
			\| date = 2012-12-10 \| accessdate = 2014-09-21
			\| website = kernelnewbies.org
			}}</ref>

			=== Booting ===
	From kernel 2.6 a new type of MD device was introduced, a ] array. The device names were modified by changing '''md''' to '''md_d'''. The partitions were identified by adding '''p''n'''''; thus '''/dev/md/md_d2p3''' for example.
			Since support for MD is found in the kernel, there is an issue with using it before the kernel is running. Specifically it ''will not'' be present if the boot loader is either ] or ] legacy. Although normally present, it ''may'' not be present for GRUB 2. In order to circumvent this problem a {{mono\|]}} filesystem must be used either without md support, or else with RAID1. In the latter case the system will boot by treating the RAID1 device as a normal filesystem, and once the system is running it can be remounted as md and the second disk added to it. This will result in a catch-up, but {{mono\|/boot}} filesystems are usually small.

			With more recent bootloaders it is possible to load the MD support as a kernel module through the ] mechanism. This approach allows the {{mono\|/boot}} filesystem to be inside any RAID system without the need of a complex manual configuration.
	From kernel 2.6.28 non-partitionable arrays can be partitioned, the partitions being referred to in the same way as for partitionable arrays: '''/dev/md/md1p2'''.

			=== {{anchor\|EXTERNAL-METADATA}}External metadata ===
	===Booting===
			Besides its own formats for RAID volumes metadata, Linux software RAID also supports external metadata formats, since version 2.6.27 of the Linux kernel and version 3.0 of the {{mono\|mdadm}} userspace utility. This allows Linux to use various firmware- or driver-based RAID volumes, also known as "]".<ref name="linux-kernel-raid">{{cite web \| url = https://raid.wiki.kernel.org/index.php/RAID_setup#External_Metadata \| title = External Metadata \| work = RAID Setup \| date = 2013-10-05 \| accessdate = 2014-01-01 \| publisher = kernel.org }}</ref>

			{{As of\|2013\|10}}, there are two supported formats of the external metadata:
	Since support for MD is found in the kernel, there is an issue with using it before the kernel is running. Specifically it ''will not'' be present if the boot loader is either ] or ] legacy. It ''may'' not be present for GRUB 2. In order to circumvent this problem a ] filesystem must be used either without md support, or else with RAID1. In the latter case the system will boot by treating the RAID1 device as a normal filesystem, and once the system is running it can be remounted as md and the second disk added to it. This will result in a catch-up, but /boot filesystems are usually small.

			* DDF (]), an industry standard defined by the ] for increased interoperability.<ref>{{cite web \| url = https://raid.wiki.kernel.org/index.php/DDF_Fake_RAID \| title = DDF Fake RAID \| work = RAID Setup \| date = 2013-09-12 \| accessdate = 2014-01-01 \| publisher = kernel.org }}</ref>
	With more recent bootloaders it is possible to load the MD support as a kernel module through the initramfs mechanism, this approach allows you to let the /boot filesystem be inside any RAID system without the need of a complex manual configuration.
			* Volume metadata format used by the ](RST), former Intel Matrix RAID, implemented on many consumer-level motherboards.<ref name="linux-kernel-raid" />

	==~~Quick~~ ~~reference~~==		=== mdmpd ===
			'''mdmpd''' was<ref>{{Cite web\|url=https://bugzilla.redhat.com/show_bug.cgi?id=117498\|title = 117498 – md code missing event interface}}</ref> a ] used for monitoring MD multipath devices up to ] 2.6.10-rc1, developed by ] as part of the mdadm package.<ref>{{cite web \| url = https://rhn.redhat.com/errata/RHEA-2003-397.html \| title = Updated mdadm package includes multi-path device enhancements \| date = 2004-01-16 \| work = RHEA-2003:397-06 \| publisher = ]}}</ref> The program was used to monitor ] (]) devices, and is usually started at boot time as a service, and afterwards running as a daemon.

			Enterprise storage requirements often include the desire to have more than one way to talk to a single disk drive so that in the event of some failure to talk to a disk drive via one controller, the system can automatically switch to another controller and keep going. This is called multipath disk access. The linux kernel implements multipath disk access via the software RAID stack known as the md (Multiple Devices) driver. The kernel portion of the md multipath driver only handles routing I/O requests to the proper device and handling failures on the active path. It does not try to find out if a path that has previously failed might be working again. That's what this daemon does. Upon startup, it reads the current state of the md raid arrays, saves that state, and then waits for the kernel to tell it something interesting has happened. It then wakes up, checks to see if any paths on a multipath device have failed, and if they have then it starts to poll the failed path once every 15 seconds until it starts working again. Once it starts working again, the daemon will then add the path back into the multipath md device it was originally part of as a new spare path.
	===Create an array===

			If one is using the {{mono\|/]}} filesystem, {{mono\|/proc/mdstat}} lists all active md devices with information about them. Mdmpd requires this to find arrays to monitor paths on, to get notification of interesting events and to monitor array reconstruction on Monitor mode.<ref>{{Cite web\|url=https://linux.die.net/man/8/mdadm\|title=Mdadm(8): Manage MD devices aka Software RAID - Linux man page}}</ref>
	mdadm --create /dev/md0 --level=mirror --raid-devices=2 /dev/sda1 /dev/sdb1

			=== Technical details RAID 1 ===
	Create a RAID 1 (mirror) array from two partitions. If the partitions differ in size, the array is the size of the smallest partition. You
			{{unsourced section\|date=November 2023}}
	can create a RAID 1 array with more than two devices. This gives you multiple copies. Whilst
			The data on a RAID 1 volume is the same as on a normal partition. The RAID information is stored in the last 128kB of the partition. This means, to convert a RAID 1 volume to normal data partition, it is possible to decrease the partition size by 128kB and change the partition ID from fd to 83 (for Linux).
	there is little extra safety in this, it makes sense when you are creating a RAID 5 array for most of your disk space and using RAID 1 only for
	a small /boot partition. Using the same partitioning for all member drives keeps things simple.

			== See also ==
	mdadm --create /dev/md1 --level=5 --raid-devices=3 /dev/sda2 /dev/sdb2 /dev/sdc2
			{{Portal\|Free and open-source software\|Linux}}
			* ] on OpenBSD/NetBSD

			== References ==
	Create a RAID 5 volume from three partitions. If the partitions used in your RAID array are not the same size, '''mdadm''' will use the size of the smallest from each partition. If you receive an error, such as: "mdadm: RUN_ARRAY failed: Invalid argument",
			{{Reflist}}
	make sure your kernel supports (either via a module or by being directly compiled in) the raid mode you are trying to use. (Most modern kernels do.)

			== External links ==
	It is possible to create a degraded mirror, with one half missing by replacing a drive name with "missing":
			{{Wikiversity\|mdadm quick reference}}
	mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sdb1 missing

	The other half mirror is added to the set thus:
	mdadm --manage /dev/md1 --add /dev/sda1

	This is useful when you are adding a disk to a computer which currently isn't mirrored. The new drive is...
	* partitioned to match the first (unless you are also repartitioning too)
	* turned into a set of "half-mirrors"
	* formatted with appropriate file system
	* mounted
	* the data is copied over,
	* made bootable
	* its grub config and fstab mounts changed
	The computer is then booted off the secondary drive (or a rescue disk), the now idle original disk can be repartitioned if required (no need to format), and then the primary drive submirrors are added.

	Note that the partition types should be changed to 0xFD with fdisk to indicate that they are mirrored devices.

	===Recording the array===
	mdadm --detail /dev/md0

	View the status of the multi disk array '''md0'''.

	mdadm -Es \| grep md >>/etc/mdadm/mdadm.conf

	This adds '''md0''' to the configuration file so that it is recognised next time you boot.

	You may wish to keep a copy of '''/proc/mdstat''' on another machine or as a paper copy. The information will allow you to restart the array
	manually if mdadm fails to do so.

	===Growing an array by adding devices===

	mdadm --add /dev/md1 /dev/sdd1
	mdadm --grow /dev/md1 --raid-devices=4

	This adds the new device to the array then grows the array to use its space.

	In some configurations you may not be able to grow the array until you have removed the internal bitmap. You can add the bitmap back again after the array has been grown.
	mdadm --grow /dev/md1 -b none
	mdadm --grow /dev/md1 -b internal

	===Growing an array by upgrading devices===

	An array may be upgraded by replacing the devices one by one, either as a planned upgrade or ad hoc as a result of replacing failed devices.

	mdadm /dev/md1 --fail /dev/sda1
	# replace the first drive with the new, larger one then partition it
	mdadm --add /dev/md1 /dev/sda1

	Allow the new drive to resync. If replacing all the devices repeat the above for each device, allowing the array to resync between repetitions. Finally, grow the array to use the maximum space available and then grow the filesystem(s) on the RAID array to use the new space.

	mdadm --grow /dev/md1 --size=max

	===Deleting an array===
	mdadm --stop /dev/md0 # to halt the array
	mdadm --remove /dev/md0 # to remove the array
	mdadm --zero-superblock /dev/sd1 # delete the superblock from all drives in the array
	# edit /etc/mdadm/mdadm.conf to delete any rows related to deleted array

	===Convert an existing partition to RAID 5===
	Assume that the existing data is on /dev/sda1:

	mdadm --create /dev/md1 --level=5 --raid-devices=3 missing /dev/sdb2 /dev/sdc2
	mdadm -Es >>/etc/mdadm/mdadm.conf
	update-initramfs -u
	dd if=/dev/sda of=/dev/md1
	# replace /dev/sdaX with /dev/md1pX in your boot loader's menu
	# reboot into /dev/md1
	# format /dev/sdaY by marking the partition as autoraid
	mdadm --add /dev/md1 /dev/sdaY
	# update your boot loader

	Notes:
	# A partition may be given as '''missing''' to act as a placeholder so that it can be added later.
	# The /boot directory should be elsewhere, possibly on /dev/md0 or its own partition.
	# If the reboot fails, do '''NOT''' add /dev/sda1 into the array until the problem is corrected!

	==Mdmpd==
	'''Mdmpd''' is a ] for the ]/] ]. It is part of the mdadm package written and copyrighted by ].<ref>{{cite web
	\| url=https://rhn.redhat.com/errata/RHEA-2003-397.html
	\| title=Updated mdadm package includes multi-path device enhancements
	\| date=2004-01-16
	\| work=RHEA-2003:397-06
	\| publisher=]}}</ref> The program is used to monitor multi-path (]) devices, and is usually started at boot time as a service, and afterwards running as a ].

	mdmpd – daemon to monitor MD multipath devices

	Enterprise storage requirements often include the desire to have more than one way to talk to a single disk drive so that in the event of some failure to talk to a disk drive via one controller, the system can automatically switch to another controller and keep going. This is called multipath disk access. The linux kernel implements multipath disk access via the software RAID stack known as the md (Multiple Devices) driver. The kernel portion of the md multipath driver only handles routing I/O requests to the proper device and handling failures on the active path. It does not try to find out if a path that has previously failed might be working again. That's what this daemon does. Upon startup, the daemon will fork and place itself in the background. Then it reads the current state of the md raid arrays, saves that state, and then waits for the kernel to tell it something interesting has happened. It then wakes up, checks to see if any paths on a multipath device have failed, and if they have then it starts to poll the failed path once every 15 seconds until it starts working again. Once it starts working again, the daemon will then add the path back into the multipath md device it was originally part of as a new spare path.

	If one is using the /proc filesystem, /proc/mdstat lists all active md devices with information about them. Mdmpd requires this to find arrays to monitor paths on and to get notification of interesting events.

	==Known problems==

	A common error when creating RAID devices is that the dmraid-driver has taken control of all the devices that are to be used in the new RAID device. Error-messages like this will occur:

	mdadm: Cannot open /dev/sdb1: Device or resource busy

	Typically, the solution to this problem involves adding the "nodmraid" kernel parameter to the boot loader config. Another way this error can present itself is if the device mapper has its way with the drives. Issue 'dmsetup table' see if the drive in question is listed. 'dmsetup remove <drive id>' will remove the drive from device mapper and the "Device or resource busy" error will go away as well.

	===RAID already running===

	First check if the device isn't in use in another array:
	cat /proc/mdstat

	Probably you will have to stop the array with:
	mdadm --stop /dev/<mdxxx>

	Check the /etc/mdadm/mdadm.conf file (and restart system if possible):
	vi /etc/mdadm/mdadm.conf

	Then you should be able to delete the superblock of this device:
	mdadm --zero-superblock /dev/sdxN

	Now the device shouldn't be busy any more.

	Sometimes dmraid "owns" the devices and will not let them go. There is a solution.<ref>{{cite web
	\| url=http://superuser.com/questions/102086/how-to-build-initrd-without-dmraid-driver-on-ubuntu-9-10
	\| title=How to build initrd without dmraid-driver on Ubuntu 9.10?
	\| author=Jonik
	\| date=2010-01-29}}</ref>

	===Tweaking the kernel===

	To solve this problem, you need to build a new ] without the dmraid-driver. The following command does this on a system with the "2.6.18-8.1.6.el5"-kernel:
	mkinitrd --omit-dmraid /boot/NO_DMRAID_initrd-2.6.18-8.1.6.el5.img 2.6.18-8.1.6.el5

	After this, the system has to be rebooted with the new initrd. Edit your /boot/grub/grub.conf to achieve this.

	Alternatively if you have a self customized and compiled kernel from a distro like Gentoo (the default option in gentoo) which doesn't use initrd then check kernel .config file in /usr/src/linux for the line
	# CONFIG_BLK_DEV_DM is not configured

	If the above line is set as follows:
	CONFIG_BLK_DEV_DM=y

	then you might have to disable that option, recompile the kernel, put it in /boot and finally edit grub conf file in /boot/grub. PLEASE be careful NOT to disable
	CONFIG_BLK_DEV_MD=y

	(Note the MD instead of DM) which is essential for raid to work at all!

	If both methods have not helped you then booting from live CD probably will (the below example is for starting a degraded raid-1 mirror array and adding a spare hdd to it and syncing. Creating a new one shouldn't be more difficult because the underlying problem was 'Device or resource busy' error):
	modprobe raid1
	mknod /dev/md1 b 9 1
	mknod /dev/md3 b 9 3
	mdadm --assemble /dev/md1 /dev/hda1
	mdadm --assemble /dev/md3 /dev/hda3
	mdadm --add /dev/md1 /dev/hdb1
	mdadm --add /dev/md3 /dev/hdb3

	It might be easier to try and automatically assemble the devices
	mdadm --assemble --scan

	Remember to change the corresponding md* and hd* values with the corresponding ones from your system.
	You can monitor the sync progress using:
	cat /proc/mdstat

	When the sync is done you can reboot in your Linux normally.

	===Zeroing the superblock===

	Another way to prevent the kernel autostarting the raid is to remove all the previous raid-related information from the disks before proceeding with the creation, for example:
	mdadm --stop /dev/md0
	mdadm --zero-superblock /dev/sd1

	And now the usual create, for example:
	mdadm --create /dev/md0 --level=5 --raid-devices=4 --spare-devices=0 /dev/sd1

	===Recovering from a loss of raid superblock===

	There are superblocks on the drives themselves and on the raid (apparently). If you have a power failure, hardware failure, that does not include the drives themselves, and you cannot get the raid to recover in any other way, and wish to recover the data, proceed as follows:

	Get a list of the devices in the raid in question:
	mdadm --detail /dev/md

	Result something like this:

	/dev/md127:
	Version : 1.2
	Creation Time : Sun Aug 21 23:35:28 2011
	Raid Level : raid6
	Array Size : 7814047744 (7452.06 GiB 8001.58 GB)
	Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB)
	Raid Devices : 6
	Total Devices : 4
	Persistence : Superblock is persistent
	Update Time : Sun Jan 1 11:43:17 2012
	State : clean, degraded
	Active Devices : 4
	Working Devices : 4
	Failed Devices : 0
	Spare Devices : 0
	Layout : left-symmetric
	Chunk Size : 512K
	Name : clop:1 (local to host clop)
	UUID : 7ee1e93a:1b011f80:04503b8d:c5dd1e23
	Events : 62
	Number Major Minor RaidDevice State
	0 8 33 0 active sync /dev/sdc1
	1 8 49 1 active sync /dev/sdd1
	2 8 81 2 active sync /dev/sdf1
	3 8 65 3 active sync /dev/sde1
	4 0 0 4 removed
	5 0 0 5 removed

	RaidDevice order (sdc1,sdd1,sdf1,sde1) and Chunk Size are critical

	Record all your raid member parameters:
	mdadm --examine /dev/sd1 \| egrep 'dev\|Update\|Role\|State\|Chunk Size'

	Look carefully at the Update time. If you have raid members attached to the motherboard and others attached to a raid card, and the card fails, but leaves enough members to keep the raid alive, you want to make a note of that.
	Look at Array State and Update Time. For example:

	/dev/sdc1:
	Update Time : Wed Jun 15 00:32:35 2011
	Array State : AAAA.. ('A' == active, '.' == missing)
	/dev/sdd1:
	Update Time : Thu Jun 16 21:49:27 2011
	Array State : .AAA.. ('A' == active, '.' == missing)
	/dev/sde1:
	Update Time : Thu Jun 16 21:49:27 2011
	Array State : .AAA.. ('A' == active, '.' == missing)
	/dev/sdf1:
	Update Time : Thu Jun 16 21:49:27 2011
	Array State : .AAA.. ('A' == active, '.' == missing)
	/dev/sdk1:
	Update Time : Tue Jun 14 07:09:34 2011
	Array State : ....AA ('A' == active, '.' == missing)
	/dev/sdl1:
	Update Time : Tue Jun 14 07:09:34 2011
	Array State : ....AA ('A' == active, '.' == missing)

	Devices sdc1, sdd1, sde1 and sdf1 are the last members in the array and will rebuild correctly. sdk1 and sdl1 left the array (in my case due to a raid card failure).

	Also note the raid member, starting with 0, the raid needs to be rebuilt in the same order.
	Chunk size is also important.

	Zero the drive superblocks

	mdadm --stop /dev/md0 # to halt the array
	mdadm --remove /dev/md0 # to remove the array
	mdadm --zero-superblock /dev/sd1

	Reassemble the raid

	mdadm --create /dev/md1 --chunk=4096 --level=6 --raid-devices=6 /dev/sdc1 /dev/sdd1 /dev/sdf1 /dev/sde1 missing missing

	'missing' tell the create command to rebuild the raid in a degraded state. sdk1 and sdl1 can be added later

	Edit /etc/mdadm.conf and add an ARRAY line with a UUID. First get the UUID for your raid:

	mdadm -D /dev/md<x>

	then:

	nano /etc/mdadm.conf

	and add something similar to the file (notice there is no # in front of the active line you are adding

	#ARRAY /dev/md0 UUID=3aaa0122:29827cfa:5331ad66:ca767371
	#ARRAY /dev/md1 super-minor=1
	#ARRAY /dev/md2 devices=/dev/hda1,/dev/hdb1
	ARRAY /dev/md1 UUID=7ee1e93a:1b011f80:04503b8d:c5dd1e23

	Save with <Ctrl-o>

	Last, mark the array possilbly dirty with:
	mdadm --assemble /dev/md1 --update=resync

	Monitor the rebuild with

	watch -n 1 cat /proc/mdstat

	All your data should be recovered!

	===Increasing RAID ReSync Performance===

	In order to increase the resync speed, we can use a bitmap, which mdadm will use to mark which areas may be out-of-sync.
	Add the bitmap with the grow option like below :
	mdadm -G /dev/md2 --bitmap=internal

	:'''Note''': mdadm – v2.6.9 – 10 March 2009 on Centos 5.5 requires this to be run on a stable "clean" array. If the array is rebuilding the following error will be displayed:

	::mdadm: failed to set internal bitmap.

	:And the following line is added to the log file:
	::md: couldn't update array info. -16

	then verify that the bitmap was added to the md2 device using
	cat /proc/mdstat

	you can also adjust Linux kernel limits by editing these files
	/proc/sys/dev/raid/speed_limit_min
	and
	/proc/sys/dev/raid/speed_limit_max

	You can also edit this with the sysctl utility
	sysctl -w dev.raid.speed_limit_min=50000

	===Increasing RAID5 Performance===
	To help RAID5 read/write performance, setting the read-ahead & stripe cache size<ref>{{cite web
	\| url=http://oss.sgi.com/archives/xfs/2007-01/msg00559.html
	\| title=Linux Software RAID 5 Performance Optimizations: 2.6.19.1: (211MB/s read & 195MB/s write)
	\| author=Justin Piszcz
	\| date=2007-01-11}}</ref>
	<ref>{{cite web
	\| url=http://www.mythtv.org/LVM_on_RAID#Performance_Enhancements_.2F_Tuning
	\| title=LVM on RAID: Performance Enhancements / Tuning
	\| date=2010-10-10
	\| accessdate=2013-05-07}}</ref> for the array provides noticeable speed improvements.

	'''Note''': This tip assumes sufficient RAM availability to the system. Insufficient RAM can lead to data loss/corruption

	echo 16384 > /sys/block/md0/md/stripe_cache_size
	blockdev --setra 16384 /dev/md0

	Write Performance:
	dd if=/dev/zero of=/mnt/10gb.16384k.stripe.out bs=1M count=10240
	10240+0 records in
	10240+0 records out
	10737418240 bytes (11 GB) copied, 94.5423 s, 114 MB/s

	Read Performance:
	dd if=/mnt/10gb.16384k.stripe.out of=/dev/null bs=1M
	10240+0 records in
	10240+0 records out
	10737418240 bytes (11 GB) copied, 28.5435 s, 376 MB/s

	These changes must be done on any reboot (add to an init script to set on start-up)

	==References==
	{{Reflist}}

			*
	==External links==
			* {{cite web \| url = https://help.ubuntu.com/community/Installation/SoftwareRAID \| title = Installation/SoftwareRAID \| work = ] Community Documentation \| date = 2012-03-01}}
	*
			* {{cite web \| url = http://www.technotes.se/?p=1732 \| title = Setting up a RAID volume in Linux with >2TB disks \| author = Lonezor \| date = 2011-11-13 \| url-status = dead \| archiveurl = https://web.archive.org/web/20111119090413/http://www.technotes.se/?p=1732 \| archivedate = 2011-11-19 }}
	*
	* {{cite web
	\| url=http://git.debian.org/?p=pkg-mdadm/mdadm.git;a=blob;f=debian/README.recipes;hb=HEAD
	\| title=mdadm recipes
	\| first=Martin F.
	\| last=Krafft
	\| work=]}}
	* {{cite web
	\| url=http://www.linuxhomenetworking.com/index.php/Quick_HOWTO_:_Ch26_:_Linux_Software_RAID
	\| title=Quick HOWTO: Ch26: Linux Software RAID
	\| publisher=linuxhomenetworking.com}}
	* {{cite web
	\| url=https://help.ubuntu.com/community/Installation/SoftwareRAID
	\| title=Installation/SoftwareRAID
	\| work=] Community Documentation
	\| date=2012-03-01}}
	* {{cite web
	\| url=http://www.technotes.se/?p=1732
	\| title=Setting up a RAID volume in Linux with >2TB disks
	\| author=Lonezor
	\| date=2011-11-13}}

	]		]

Latest revision as of 08:12, 11 June 2024

Linux utility for managing software RAID

This article includes a list of general references, but it lacks sufficient corresponding inline citations. Please help to improve this article by introducing more precise citations. (November 2011) (Learn how and when to remove this message)

mdadm
Original author(s)	Neil Brown
Developer(s)	Community contributors, Mariusz Tkaczyk
Initial release	2001

Stable release	4.3 / February 15, 2024; 10 months ago (2024-02-15)

Repository	github.com/md-raid-utilities/mdadm/
Written in	C
Operating system	Linux
Available in	English
Type	Disk utility
License	GNU GPL
Website	raid.wiki.kernel.org

mdadm is a Linux utility used to manage and monitor software RAID devices. It is used in modern Linux distributions in place of older software RAID utilities such as raidtools2 or raidtools.

mdadm is free software originally maintained by, and copyrighted to, Neil Brown of SUSE, and licensed under the terms of version 2 or later of the GNU General Public License.

Name

The name is derived from the md (multiple device) device nodes it administers or manages, and it replaced a previous utility mdctl. The original name was "Mirror Disk", but was changed as more functions were added. The name is now understood to be short for Multiple Disk and Device Management.

Overview

Linux software RAID configurations can include anything presented to the Linux kernel as a block device. This includes whole hard drives (for example, /dev/sda), and their partitions (for example, /dev/sda1).

RAID configurations

Main article: Standard RAID levels See also: Linux MD RAID 10

RAID 0 – Block-level striping. MD can handle devices of different lengths, the extra space on the larger device is then not striped.
RAID 1 – Mirror.
RAID 4 – Like RAID 0, but with an extra device for the parity.
RAID 5 – Like RAID 4, but with the parity distributed across all devices.
RAID 6 – Like RAID 5, but with two parity segments per stripe.
RAID 10 – Take a number of RAID 1 mirrorsets and stripe across them RAID 0 style.

RAID 10 is distinct from RAID 0+1, which consists of a top-level RAID 1 mirror composed of high-performance RAID 0 stripes directly across the physical hard disks. A single-drive failure in a RAID 10 configuration results in one of the lower-level mirrors entering degraded mode, but the top-level stripe performing normally (except for the performance hit). A single-drive failure in a RAID 0+1 configuration results in one of the lower-level stripes completely failing, and the top-level mirror entering degraded mode. Which of the two setups is preferable depends on the details of the application in question, such as whether or not spare disks are available, and how they should be spun up.

Non-RAID configurations

Linear – concatenates a number of devices into a single large MD device – (deprecated since 2021 and removed from the Linux kernel since 2023 )
Multipath – provides multiple paths with failover to a single device
Faulty – a single device which emulates a number of disk-fault scenarios for testing and development
Container – a group of devices managed as a single device, in which one can build RAID systems

Features

The original (standard) form of names for md devices is /dev/md<n>, where <n> is a number between 0 and 99. More recent kernels have support for names such as /dev/md/Home. Under 2.4.x kernels and earlier these two were the only options. Both of them are non-partitionable.

Since 2.6.x kernels, a new type of MD device was introduced, a partitionable array. The device names were modified by changing md to md_d. The partitions were identified by adding p<n>, where <n> is the partition number; thus /dev/md/md_d2p3 for example. Since version 2.6.28 of the Linux kernel mainline, non-partitionable arrays can be partitioned, the partitions being referred to in the same way as for partitionable arrays – for example, /dev/md/md1p2.

Since version 3.7 of the Linux kernel mainline, md supports TRIM operations for the underlying solid-state drives (SSDs), for linear, RAID 0, RAID 1, RAID 5 and RAID 10 layouts.

Booting

Since support for MD is found in the kernel, there is an issue with using it before the kernel is running. Specifically it will not be present if the boot loader is either (e)LiLo or GRUB legacy. Although normally present, it may not be present for GRUB 2. In order to circumvent this problem a /boot filesystem must be used either without md support, or else with RAID1. In the latter case the system will boot by treating the RAID1 device as a normal filesystem, and once the system is running it can be remounted as md and the second disk added to it. This will result in a catch-up, but /boot filesystems are usually small.

With more recent bootloaders it is possible to load the MD support as a kernel module through the initramfs mechanism. This approach allows the /boot filesystem to be inside any RAID system without the need of a complex manual configuration.

External metadata

Besides its own formats for RAID volumes metadata, Linux software RAID also supports external metadata formats, since version 2.6.27 of the Linux kernel and version 3.0 of the mdadm userspace utility. This allows Linux to use various firmware- or driver-based RAID volumes, also known as "fake RAID".

As of October 2013, there are two supported formats of the external metadata:

DDF (Disk Data Format), an industry standard defined by the Storage Networking Industry Association for increased interoperability.
Volume metadata format used by the Intel Rapid Storage Technology(RST), former Intel Matrix RAID, implemented on many consumer-level motherboards.

mdmpd

mdmpd was a daemon used for monitoring MD multipath devices up to Linux kernel 2.6.10-rc1, developed by Red Hat as part of the mdadm package. The program was used to monitor multipath (RAID) devices, and is usually started at boot time as a service, and afterwards running as a daemon.

Enterprise storage requirements often include the desire to have more than one way to talk to a single disk drive so that in the event of some failure to talk to a disk drive via one controller, the system can automatically switch to another controller and keep going. This is called multipath disk access. The linux kernel implements multipath disk access via the software RAID stack known as the md (Multiple Devices) driver. The kernel portion of the md multipath driver only handles routing I/O requests to the proper device and handling failures on the active path. It does not try to find out if a path that has previously failed might be working again. That's what this daemon does. Upon startup, it reads the current state of the md raid arrays, saves that state, and then waits for the kernel to tell it something interesting has happened. It then wakes up, checks to see if any paths on a multipath device have failed, and if they have then it starts to poll the failed path once every 15 seconds until it starts working again. Once it starts working again, the daemon will then add the path back into the multipath md device it was originally part of as a new spare path.

If one is using the /proc filesystem, /proc/mdstat lists all active md devices with information about them. Mdmpd requires this to find arrays to monitor paths on, to get notification of interesting events and to monitor array reconstruction on Monitor mode.

Technical details RAID 1

This section does not cite any sources. Please help improve this section by adding citations to reliable sources. Unsourced material may be challenged and removed. (November 2023) (Learn how and when to remove this message)

The data on a RAID 1 volume is the same as on a normal partition. The RAID information is stored in the last 128kB of the partition. This means, to convert a RAID 1 volume to normal data partition, it is possible to decrease the partition size by 128kB and change the partition ID from fd to 83 (for Linux).

References

"Announcement: mdadm maintainer update". marc.info. 2023-12-14. Retrieved 2024-05-17.
"Release mdadm-4.3". 2024-02-15.
^ Bresnahan, Christine; Blum, Richard (2016). LPIC-2: Linux Professional Institute Certification Study Guide. John Wiley & Sons. pp. 206–221. ISBN 9781119150817.
Vadala, Derek (2003). Managing RAID on Linux. O'Reilly Media, Inc. p. 140. ISBN 9781565927308. mdadm linux.
Nemeth, Evi (2011). UNIX and Linux System Administration Handbook. Pearson Education. pp. 242–245. ISBN 9780131480056.
"Mdadm". Archived from the original on 2013-05-03. Retrieved 2007-08-25.
"md: Remove deprecated CONFIG_MD_LINEAR". GitHub. 2023-12-19. Retrieved 2024-04-30.
"Linux kernel 3.7, Section 5. Block". kernelnewbies.org. 2012-12-10. Retrieved 2014-09-21.
^ "External Metadata". RAID Setup. kernel.org. 2013-10-05. Retrieved 2014-01-01.
"DDF Fake RAID". RAID Setup. kernel.org. 2013-09-12. Retrieved 2014-01-01.
"117498 – md code missing event interface".
"Updated mdadm package includes multi-path device enhancements". RHEA-2003:397-06. Redhat. 2004-01-16.
"Mdadm(8): Manage MD devices aka Software RAID - Linux man page".

External links

mdadm source code releases
"Installation/SoftwareRAID". Ubuntu Community Documentation. 2012-03-01.
Lonezor (2011-11-13). "Setting up a RAID volume in Linux with >2TB disks". Archived from the original on 2011-11-19.

Categories: