The maximum number of SCSI disks supported by Linux

Questions:
I have been thinking about this title for a long time? I thought about the following titles,
A: What is the maximum number of SCSI Luns supported by Linux?
B: What is the maximum number of SAN storage disks supported by Linux?
After checking for N laps, I decided to use the title now. As for why I will give the answer later.

Actual situation:
The problem was originally simple. On that day, the customer called to expand the LVM. At that time, I thought about the environment. It was nothing more than storing the transferred disks. I implemented it using the multipath software multipath-tools that comes with SuSE through two license cards. The process is nothing more than controlling the HA environment, setting PV, adding it to VG, and then expanding LV. Once again confirmed with the customer, the customer stressed that two storage disks were used this time. Call the boss to ask if the two storage services will affect the performance. The boss said no. So I answered the customer's OK and there was no problem. We will discuss the details over Friday. At that time, we communicated about the scale-up to 2.5 TB. Currently, it is only about GB. The size of a single disk is only 15 GB. Think about it. Then go back and check the information to confirm how many devices are supported. At that time, I didn't do a math problem. I figured out how many disks it was. This was a big failure, and it finally became the starting point of this event. Although there are no specific conversions, I am aware that we must determine the number of SCSI Luns supported in the system. So we started the query with this problem.

Reference Links and event processes:
[Kook written aboveFamous saying: No matter how official a link is, do not trust it easily. This is my failure lesson. If you ask me what I should believe, go to the source code.]
To determine how many storage devices Linux supports, I started my Google journey. Use keywords such as "Linux Maximum SCSI disk Luns multiple scsi_mod sd_mod" to match multiple times. Find the following links in sequence.

Phase I: Error phase (Keyword: Linux support maximum Luns)
A: http://kbase.redhat.com/faq/docs/DOC-3293
Have a look Article To modify the module scsi_mod, add max_luns = 512, redo the guided initrd image, and add the module parameter changes to initrd. This is certainly no problem. This can be done.

B: http://download.qlogic.com/sf/62552/ibm_dd_ds4kfc1_8.01.06_linux2.6_anycpu.txt
I looked at it and found a bright section of my eyes.
7.1 enable more than 1 SCSI device per Adapter
Support for multiple Luns per device is standard in Linux 2.6 kernels. Currently, the maximum number of Luns that can be scanned for each device is 256.

C: source code file/usr/src/Linux/Drivers/SCSI/scsi_scan.c
# Define max_scsi_luns 512

After seeing the reference link above, I feel that this is not completely correct. With this conclusion, I went to the customer to scale up the four groups of HA environments. First, the group with the largest capacity is eliminated. During the expansion, lsscsi recognized 330 disks and 165 disks after multipath processing. At that time, it was still smooth, that is, it was found that the system had been restarted for too long. It takes about 10 minutes to stop at the devices-mapper during self-check. The entire boot time is less than 30 minutes. It was rough. The system has not been restarted for too long, and the file system has passed the self-check. That's why I didn't mean that much. In fact, the system has given me a warning. In the three groups that follow, the number of disks is less than 100, so it is easy to learn and complete very quickly. Go out at one o'clock P.M. a.m. and start at a.m.. At a.m., you receive the support logs of the host status after resizing and receive the logs of one of the hosts in the Lun group at most, the bastion host times out and the script is interrupted. As a result, the host fails to respond, and the HA controls its automatic restart. After the system is restarted, check that the system is running normally. It's already four o'clock back home.

The customer received a call at pm. The system was abnormal. Some Application prompts that some files were not completely sent. From the customer's description, it should be an IO exception, resulting in incomplete data files. Open the log to view it. I rely on it. It's really not normal. Lsscsi outputs 330 SD * files, but fdisk-l outputs only 256 disk devices. See the underlying device with multi-path multipath. Some two paths are normal, and some have only one path. Count and add 256. Crash. Try the following at the site:

A: added kernel boot parameters for grub.
Max_scsi_runs = 512
Result: The system is restarted and the problem persists.

B: Modify the scsi_mod parameter. max_luns = 512

The procedure is as follows:
1: Modify/etc/modprobe. conf. Local and add
Options scsi_mod max_luns = 512
2. Create an image
/Sbin/mk_initrd
3: Check the image, decompress the image, and check the output as follows.
Grep scsi_mod linuxrc

Echo "loading kernel/Drivers/SCSI/scsi_mod.ko $ extra_scsi_params max_luns = 512"
Insmod/lib/modules/2.6.5-7.315-bigsmp/kernel/Drivers/SCSI/scsi_mod.ko $ extra_scsi_params max_luns = 512

Result: The system is restarted and the problem persists.

The customer can't wait any longer. After a while, it's time to process the business. After three o'clock, the customer decided to reduce the capacity to less than 256 that the system could recognize. Then, copy the backup file, delete LV, VG, PV, and restart the system. Store and repartition the disk. Rebuild LVM and restore the file. An error occurred during VG reconstruction, a change in the NIC loading sequence, and an error occurred while the network was disconnected. It's a coincidence. Fortunately, the system was restored before dawn. The system runs normally when you sleep at the customer's hotel in the afternoon. So I took a taxi home.

As a matter of fact, it is not over yet. When I prepare to go to bed again at, the customer's phone will ring again. At that time, I felt like a crash! This will not be written into it later. I will write another log to write the following things.

Stage II: analysis stage (Keyword: Linux scsi_mod sd_mod devices SCSI disk maximum)
When I calmed down and carefully looked at the logs, I checked the system files carefully. The cloud was gradually opened at, but I still did not understand it in some places.
1: The system can see 330 disks.
A: lsscsi shows 330 SD * devices.
B: The number of SD * devices in CAT/proc/partition is 330.
C: CAT/proc/SCSI/qla2xxx/0, 1 detected 165 devices respectively

2. The system can use 256 Devices
A: The fdisk-L output block device is 256
B: lsmod indicates that sd_mod is used as 256.
In the C: multipath-ll output, the block device actually used is 256.

Why?
1: Why is lsscsi recognizable?
Http://sg.danny.cz/scsi/lsscsi.html
2: why there are only 256 sd_mod in lsmod.

Next, let's take a look at the reason. The SCSI is essentially a physical SCSI bus that communicates with peripherals inside a computer, a scsi device is a device that adopts the SCSI bus standard. Now, what is the underlying support in Linux? To view. The most basic module is scsi_mod. What does it mean? All SCSI bus-compliant devices need to call this module. As for the modules that depend on him, you can see that CCISS is an HP hard disk, St is a SCSI tape, and qla2xxx is a Qlogic memory card, sd_mod is the SCSI disk device SD * we see.

[Root @ server/] # lsmod
Module size used
SG 54176 0
St 57500 0
Sr_mod 33316 0
CDROM 55196 2 sr_mod, ide_cd
Dm_round_robin 19968 1
Dm_multipath 37512 2 dm_round_robin
Dm_mod 73600 46 dm_multipath
Qla2300 141056 0
Qla2xxx 365496 169 qla2300
Firmware_class 25984 1 qla2xxx
Scsi_transport_fc 23296 1 qla2xxx
CCISS 73796 7
Sd_mod 37888 256
Scsi_mod 134852 8 osst, SG, St, sr_mod, qla2xxx, scsi_transport_fc, CCISS, sd_mod

#/Sbin/modinfo scsi_mod
Parm: scsi_logging_level: A bit mask of logging levels (INT)
License: GPL
Description: SCSI Core
Parm: inq_timeout: timeout (in seconds) waiting for devices to answer inquiry. Default is 5. Some non-compliant devices need more. (INT)
Parm: max_report_luns: Report Luns maximum number of Luns encoded ed (shocould be between 1 and 16384) (INT)
Parm: max_luns: Last SCSI Lun (shocould be between 1 and 2 ^ 32-1) (INT)
Parm: default_dev_flags: SCSI default device flag integer value (INT)
Parm: dev_flags: Given scsi_dev_flags = vendor: Model: Flags [, V: M: f] Add black/white list entries for vendor and model with an integer value of flags to the SCSI device info list (string)
Depends:
Supported: Yes
Vermagic: 2.6.5-7.246-bigsmp SMP PentiumII regparm gcc-3.3

#/Sbin/modinfo sd_mod
Description: SCSI disk (SD) Driver
Author: Eric youngdale
License: GPL
Depends: scsi_mod
Supported: Yes
Vermagic: 2.6.5-7.246-bigsmp SMP PentiumII regparm gcc-3.3

Are there any clues Through the above instructions? The scsi_mod module supports all devices that support the SCSI bus on the hardware and is the underlying driver, but we want to know that the sd_mod module that supports the most SCSI disks in Linux does not contain any parameters. Rely on me! Let's take a look at the source code.

View the source code file of the sd_mod module:/usr/src/Linux/Drivers/SCSI/SD. c

/* Device no to disk mapping:
*
* Major disc2 disc p1
* | ............ | ............. | .... | .... | <-Dev_t
* 31 20 19 8 7 4 3 0
*
* Inside a major, We have 16 K disks, however mapped non-
* Contiguously. the first 16 disks are for major0, the next
* Ones with major1,... disk 256 is for major0 again, disk 272
* For major1 ,...
* As we stay compatible with our numbering scheme, we can reuse
* The well-know SCSI majors 8, 65--71, 136--143.
*/

Static int sd_major (INT major_idx)
{
Switch (major_idx ){
Case 0:
Return scsi_disk0_major;
Case 1... 7:
Return scsi_disk1_major + major_idx-1;
Case 8... 15:
Return scsi_disk8_major + major_idx-8;
Default:
Bug ();
Return 0;/* shut up GCC */
}
}
Find this and finally find out why only 256 files are available. View the kernel documentation/usr/src/Linux/documentation/devices.txt, find the definition section of major for 8, 65-71,136-143, and keep it for SCSI disk.

 

Check the output () in the memory (). About the bolock section of the file/proc/devices.
Block devices:
8 SD
65 SD
66 SD
67 SD
68 SD
69 SD
70 SD
71 SD
128 SD
129 SD
130 SD
131 SD
132 SD
133 SD
134 SD
135 SD
Check whether there are 16 major files. What does this mean? See the definitions in/usr/src/Linux/documentation/devices.txt. Major is 8, 65-71,136-143 (16 in total), while minor can use 256. The limit of a partition is 16, which is actually 15.

The following is the conclusion: How many SCSI disks can be used in Linux? That is, Major * minor/Partitions = 16*256/16 = 256. Therefore, for disks that have been stored in a row, because two HbA cards are used, 165x2 = 330 disks, but the system can only use 256 disks. In this way, all the above settings are established. Due to the limitations of sd_mod, we cannot use all of its disks. The scsi_mod module is directly used as the driver of the HBA card. It can identify all the Luns, but the system cannot use them.

It seems that the process is over, but the process is not over. I Google the following link.
SCSI disk limitations on Linux
Http://publib.boulder.ibm.com/infocenter/dsichelp/ds8000ic/index.jsp? Topic =/com.ibm.storage.ssic.help.doc/f2c_linuxscsilimit_2hsag9.html

Newer kernels are moving to a dynamic configuration method that will use any available major numbers after the first 16 reserved major numbers have been used. the latest SuSE sles 8 kernel errata primary des this change and imposes a limit of no more than 144 major numbers allocated to SCSI disk devices. thus these kernels can support up to 2304 disk devices.

In general, the following formula provides the maximum number of devices for Linux host systems: number of devices = (number of major numbers) x (number of minor numbers) limit (number of partitions ). for example, number of devices = 8x256 bytes 16 = 128 or number of devices = 144x256 bytes 16 = 2304.

For the Linux 2.6 kernels the number of major and minor BITs has been increased to 12 and 20 BITs respectively, thus Linux 2.6 kernels can support thousands of disks. there is still the limitation of only up to 15 partitions per disk.

After reading this, my new question comes: How does the kernel use undefined and available major. I read the log again, skipped the other, and only extracted devices with the major value of 16, and found the following:
This is a normal device. Major is defined in sd_mod, and minor is an integer multiple of 16 and smaller than 256.
8 0 SDA
8 16 SDB
8 32 SDC
8 48 SDD
8 64 SDE
8 80 SDF
8 96 SDG
8 112 SDH
8 128 SDI
8 144 SDJ
8 160 SDK
8 176 SDL
8 192 SDM
8 208 SDN
8 224 SDO
8 240 SDP
For abnormal SD devices, see the following:
8 256 sdiw
8 272 sdix
8 288 sdiy
8 304 sdiz
8 320 sdja
8 336 sdjb
8 352 sdjc
8 368 sdjd
8 384 sdje
8 400 sdjf
8 416 sdjg
8 432 sdks
8 448 sdji
8 464 sdjj
8 480 sdjk
8 496 sdjl

I collapsed and the problem is still not over. As far as my current understanding is concerned, I cannot understand it. How are the major and minor of these invalid SD * devices allocated? The document describes how the available major retained by the kernel is allocated and used when 16 sd_mod are exceeded. How can I enable the system to use more than 256 SCSI disk devices?

Link to this article:Http://linux.52zhe.info/read.php/237.htm
Author:Kook (if you are talking about technical issues in your blog, contact me via MSN or gmail below)
Contact info:(MSN: kook # live.com) (Google Talk: kookliu)