SAS expander and the code implementation in Linux

1: Due to work, I have been engaged in the development of 2 * 36port Expander in LSI for the past few months. It mainly parses and processes the SSP, SMP, and SES command protocols in SAS.

The SDK of LSI uses the threadx operating system. The kernel of its chip is ARM920T, which provides the SAS protocol processing function at 920t. I personally feel that the Code style is not good.

My personal understanding of expander is as follows:

The lsisas2x36 is a 36-port, 6.0-Gbit/s Serial Attached SCSI (SAS) expander that enables the connection of up to 36 directly attached SAS or Serial ATA (SATA) devices, and provides table routing
Support connections for up to 1024 SAS addresses. Each expander PHY is individually writable able and performs SAS and SATA transfers based
On the speed of the host or target at either 6.0-gbits/s, 3.0-gbits/s, or 1.5-gbits/s. in the system, the expander role is in the "middle layer", that is, the initiator is connected up, and the target is connected down. The above description shows that the 36port expander can directly connect 36 TSAs or SATA targets. Expander uses the routing mechanism when performing progressive data or command forwarding. Route data packets based on the SAS address of each target.

Supported protocols:

A serial SCSI protocol (SSP)
B Serial ATA tunneled protocol (STP)
C serial Management Protocol (SMP)
D SAS protocol, described in the Serial Attached SCSI (SAS) standard, revision 15a
E sata, as defined in the Serial ATA: High Speed serialized at Attachment specification, Version 2.5
F SATA II

H SFF-8485 Protocol, using the serial gpio (sgpio) interface provided by theexpander

For the SAS, SSP, STP, and SMP protocols mentioned above, refer to the following chapters:

SCSI architecture model (SAM-5)

SAS protocol layer (SPL)

SCSI enclosure services-3 (SES-3)

SCSI primary commands-4 (SPC-4)

SCSI block commands (SBC-3)

SFF-8485 protocol is mainly used to control gpio.

The following figure shows the role of Expander in the entire storage system (not as I drew)

As you can see, expander is in the middle layer of the system, connects initiator Up, And connects Target down. Because the target is directly connected down, the routing types in the figure is D, indicating direct.

Multiple expanders are used to increase the number of targets.

Path redundancy mainly ensures system reliability and prevents spof.

2: expander-related code in Linux

The following code is for the Code provided by Marvell, where sas88se98.0/9445 6 Gb/s SAS/SATA Io controllers is sascontroller, And the expander code comes from E: \ SRC \ linux-2.6.33.20 \ drivers \ SCSI \ libsas sas_expander.c in this directory.

Now let's take a look at the expander-related code in Linux. The directory of the SCSI and SAS-related code in Linux is as follows:

E: \ SRC \ linux-2.6.33.20 \ drivers \ SCSI

E: \ SRC \ linux-2.6.33.20 \ include \ SCSI

Where

Role of sas_expander.c Serial Attached SCSI (SAS) expander discovery and Configuration

Role of sas_init.c Serial Attached SCSI (SAS) Transport Layer Initialization

File mv_init.c

Static int _ init mvs_init (void)
{
Int RC;
Mvs_stt = sas_domain_attach_transport (& mvs_transport_ops); // The function is defined in sas_init.c.
If (! Mvs_stt)
Return-enomem;

Rc = pci_register_driver (& mvs_pci_driver );

If (RC)
Goto err_out;

Return 0;

Err_out:
Sas_release_transport (mvs_stt );
Return RC;
}

// The function is defined in sas_init.c.

Struct scsi_transport_template *
Sas_domain_attach_transport (struct sas_domain_function_template * DFT)
{
Struct scsi_transport_template * STT = sas_attach_transport (& SFT); // The SFT is defined below
Struct sas_internal * I;

If (! STT)
Return STT;

I = to_sas_internal (STT );
I-> DFT = DFT;
STT-> create_work_queue = 1;
STT-> eh_timed_out = sas_scsi_timed_out;
STT-> eh_strategy_handler = sas_scsi_recover_host;

Return STT;
}

// Function pointer definition, mainly used to obtain expander, phy, and enclosure information.

/* The functions by which the transport class and the driver communicate */
Struct sas_function_template {
INT (* get_linkerrors) (struct sas_phy *);
INT (* get_enclosure_identifier) (struct sas_rphy *, u64 *);
INT (* get_bay_identifier) (struct sas_rphy *);
INT (* phy_reset) (struct sas_phy *, INT );
INT (* phy_enable) (struct sas_phy *, INT );
INT (* set_phy_speed) (struct sas_phy *, struct sas_phy_linkrates *);
INT (* smp_handler) (struct scsi_host *, struct sas_rphy *, struct Request *);
};

Static struct sas_function_template SFT = {// initialization of the above struct
. Phy_enable = sas_phy_enable,
. Phy_reset = sas_phy_reset,
. Set_phy_speed = sas_set_phy_speed,
. Get_linkerrors = sas_get_linkerrors,
. Smp_handler = sas_smp_handler,
};

Now let's look at a simple. set_phy_speed = sas_set_phy_speed,

Sas_set_phy_speed is defined as follows:

Enum sas_linkrate {
/* These values are defined in the SAS standard */
Sas_link_rate_unknown = 0,
Sas_phy_disabled = 1,
Sas_phy_reset_problem = 2,
Sas_sata_spinup_hold = 3,
Sas_sata_port_selector = 4,
Sas_phy_reset_in_progress = 5,
Sas_link_rate_1_5_gbps = 8,
Sas_link_rate_g1 = sas_link_rate_rj5_gbps,
Sas_link_rate_3_0_gbps = 9,
Sas_link_rate_g2 = sas_link_rate_3_0_gbps,
Sas_link_rate_6_0_gbps = 10,
/* These are virtual to the transport class and may never
* Be signalled normally since the standard defined field
* Is only 4 bits */
Sas_link_rate_failed = 0x10,
Sas_phy_virtual = 0x11,
}; Enumm defined as linkrate. Defines the link rate of SAS.

Int
Sas_set_phy_speed (struct sas_phy * phy, struct sas_phy_linkrates * rates)
{
Int ret;

// Check whether the SAS speed is correct.

If (rates-> minimum_linkrate & rates-> minimum_linkrate> phy-> maximum_linkrate) |
(Rates-> maximum_linkrate & rates-> maximum_linkrate <phy-> minimum_linkrate ))
Return-einval;

If (rates-> minimum_linkrate & rates-> minimum_linkrate <phy-> minimum_linkrate_hw)
Rates-> minimum_linkrate = phy-> minimum_linkrate_hw;

If (rates-> maximum_linkrate & rates-> maximum_linkrate> phy-> maximum_linkrate_hw)
Rates-> maximum_linkrate = phy-> maximum_linkrate_hw;

If (scsi_is_sas_phy_local (PHY) {// you can determine which control function to use to execute based on the value of scsi_is_sas_phy_local (PHY ).
Struct scsi_host * shost = dev_to_shost (phy-> Dev. Parent );
Struct sas_ha_struct * sas_ha = shost_to_sas_ha (shost );
Struct asd_sas_phy * asd_phy = sas_ha-> sas_phy [phy-> number];
Struct sas_internal * I =
To_sas_internal (sas_ha-> core. shost-> transportt );

Ret = I-> DFT-> lldd_control_phy (asd_phy, phy_func_set_link_rate, rates );
} Else {
Struct sas_rphy * rphy = dev_to_rphy (phy-> Dev. Parent );
Struct domain_device * ddev = sas_find_dev_by_rphy (rphy );
Ret = sas_smp_phy_control (ddev, phy-> number,
Phy_func_link_reset, rates );

}

Return ret;
}

I-> DFT-> lldd_control_phy function pointer assignment is carried out in the following struct

Static struct sas_domain_function_template mvs_transport_ops = {
. Lldd_dev_found = mvs_dev_found,
. Lldd_dev_gone = mvs_dev_gone,

. Lldd_execute_task = mvs_queue_command,
. Lldd_control_phy = mvs_phy_control,

. Lldd_abort_task = mvs_abort_task,
. Lldd_abort_task_set = mvs_abort_task_set,
. Lldd_clear_aca = mvs_clear_aca,
. Lldd_clear_task_set = mvs_clear_task_set,
. Lldd_ I _t_nexus_reset = mvs_ I _t_nexus_reset,
. Lldd_lu_reset = mvs_lu_reset,
. Lldd_query_task = mvs_query_task,

. Lldd_port_formed = mvs_port_formed,
. Lldd_port_deformed = mvs_port_deformed,

};

Mvs_phy_control is defined as follows:

Int mvs_phy_control (struct asd_sas_phy * sas_phy, Enum phy_func func, void * funcdata)
{
Int rc = 0, phy_id = sas_phy-> ID;
U32 TMP, I = 0, hi;
Struct sas_ha_struct * Sha = sas_phy-> HA;
Struct mvs_info * MVI = NULL;

While (Sha-> sas_phy [I]) {
If (Sha-> sas_phy [I] = sas_phy)
Break;
I ++;
}
Hi = I/(struct mvs_prv_info *) Sha-> lldd_ha)-> n_phy;
MVI = (struct mvs_prv_info *) Sha-> lldd_ha)-> MVI [HI];

Switch (func ){
Case phy_func_set_link_rate:
Mvs_chip_disp-> phy_set_link_rate (MVI, phy_id, funcdata );
Break;

Case phy_func_hard_reset:
TMP = mvs_chip_disp-> read_phy_ctl (MVI, phy_id );
If (TMP & phy_rst_hard)
Break;
Mvs_chip_disp-> phy_reset (MVI, phy_id, 1 );
Break;

Case phy_func_link_reset:
Mvs_chip_disp-> phy_enable (MVI, phy_id );
Mvs_chip_disp-> phy_reset (MVI, phy_id, 0 );
Break;

Case phy_func_disable:
Mvs_chip_disp-> phy_disable (MVI, phy_id );
Break;
Case phy_func_release_spinup_hold:
Default:
Rc =-eopnotsupp;
}
Msleep (200 );
Return RC;
} The above case statement is defined in the SPL's 9.4.3.28 PHY Control Function

Marvell 88se98.0/9445 6 Gb/s SAS/SATA Io Controllers

Const struct mvs_dispatch mvs_94xx_dispatch = {// defines the function in the case above.
"Mv94xx ",
Mvs_94xx_init,
Null,
Mvs_94xx_ioremap,
Mvs_94xx_iounmap,
Mvs_94xx_isr,
Mvs_94xx_isr_status,
Mvs_94xx_interrupt_enable,
Mvs_94xx_interrupt_disable,
Mvs_read_phy_ctl,
Mvs_write_phy_ctl,
Mvs_read_port_cfg_data,
Mvs_write_port_pai_data,
Mvs_write_port_cfg_addr,
Mvs_read_port_vsr_data,
Mvs_write_port_vsr_data,
Mvs_write_port_vsr_addr,
Mvs_read_port_irq_stat,
Mvs_write_port_irq_stat,
Mvs_read_port_irq_mask,
Mvs_write_port_irq_mask,
Mvs_get_sas_addr,
Mvs_94xx_command_active,
Mvs_94xx_issue_stop,
Mvs_start_delivery,
Mvs_rx_update,
Mvs_int_full,
Mvs_94xx_assign_reg_set,
Mvs_94xx_free_reg_set,
Mvs_get_prd_size,
Mvs_get_prd_count,
Mvs_94xx_make_prd,
Mvs_94xx_detect_porttype,
Mvs_94xx_oob_done,
Mvs_94xx_fix_phy_info,
Null,
Mvs_94xx_phy_set_link_rate,
Mvs_hw_max_link_rate,
Mvs_94xx_phy_disable,
Mvs_94xx_phy_enable,
Mvs_94xx_phy_reset,
Null,
Mvs_94xx_clear_active_cmds,
Mvs_94xx_spi_read_data,
Mvs_94xx_spi_write_data,
Mvs_94xx_spi_buildcmd,
Mvs_94xx_spi_issuecmd,
Mvs_94xx_spi_waitdataready,
# Ifndef disable_hotplug_dma_fix
Mvs_94xx_fix_dma,
# Endif
};

To make the image better:

 

 

 

 

 

 

 

2: sas_init.c File

At the end of this file, we can see the following functions:

Static int _ init sas_class_init (void)
{
Sas_task_cache = kmem_cache_create ("sas_task", sizeof (struct sas_task), 0, slab_hwcache_align, null); // The following describes
If (! Sas_task_cache)
Return-enomem;

Return 0;
}

Static void _ exit sas_class_exit (void)
{
Kmem_cache_destroy (sas_task_cache );
}

Complete slab cache application and release in the initialization and exit functions.

In kernel programming, some data structures may need to be used and released repeatedly. According to the general idea, kmalloc and kfree may be used for implementation.
However, this method is not efficient, and Linux provides us with a more efficient method-slab cache manager
Use the kmem_cache_create function to create a high-speed cache header pointer-in the kernel, It is the struct kmem_cache structure. The specific usage can be as follows: struct kmem_cache * caclap = NULL;

Caclap = kmem_cache_create ("cache_name", sizeof (struct yourstruct), 0, slab_hwcache_align, null, null );

In this way, we obtain an available cacheheader pointer.