[BUG] - Incompatibility with mpt3sas

[Forza-tng]

Describe the bug
I cannot update firmware on SATA Exos X18 drives on mpt3sas - Broadcom HBA 9500-8i (SAS3808)

To Reproduce
Steps to reproduce the behavior:

1. Attach Exos SATA drives

2. use openSeaChest to update firmware

3. update aborts after 30 seconds

4. See error in dmesg

Expected behavior
Firmware should have been updated, but the download fails.

Verbose Output

[2277096.910173] [ T1514] sd 0:0:2:0: task abort: SUCCESS scmd(0x000000004b102cf7)
[2277166.049222] [ T1514] sd 0:0:2:0: attempting task abort!scmd(0x0000000043239087), outstanding for 30132 ms & timeout 30000 ms
[2277166.049227] [ T1514] sd 0:0:2:0: [sdc] tag#3761 CDB: ATA command pass through(12)/Blank a1 16 03 03 00 02 00 00 00 92 00 00
[2277166.049229] [ T1514] scsi target0:0:2: handle(0x0028), sas_address(0x5003048020db4542), phy(2)
[2277166.049231] [ T1514] scsi target0:0:2: enclosure logical id(0x5003048020db457f), slot(2)
[2277166.049233] [ T1514] scsi target0:0:2: enclosure level(0x0000), connector name( C0  )
[2277166.409679] [ T1514] sd 0:0:2:0: task abort: SUCCESS scmd(0x0000000043239087)

Desktop (please complete the following information):

• OS: Alpine Linux 3.22
• Kernel 6.12.25
• openSeaChest-v25.05.1-linux-i686-portable

verboseinfo.zip

Version Info for openSeaChest_Firmware:
        Utility Version: 4.3.0
        opensea-common Version: 6.0.2
        opensea-transport Version: 10.0.0
        opensea-operations Version: 9.1.0
        Build Date: Jun  5 2025
        Compiled Architecture: X86
        Detected Endianness: Little Endian
        Compiler Used: GCC
        Compiler Version: 11.2.1
        Operating System Type: Linux
        Operating System Version: 6.12.25-0
        Operating System Name: Alpine Linux v3.21

Additional context
I have /sys/block/sdc/device/timeout and eh_timeout at 300 seconds, but it seems that mpt3sas driver uses a hard coded 30 second value, so it aborts the command before the firmware is fully updated.

In mpt3sas_scsih.cfunction scsih_abort():
https://github.com/torvalds/linux/blob/19272b37aa4f83ca52bdf9c16d5d81bdd1354494/drivers/scsi/mpt3sas/mpt3sas_scsih.c#L3292

	u8 timeout = 30;
	struct _pcie_device *pcie_device = NULL;
	sdev_printk(KERN_INFO, scmd->device, "attempting task abort!"
	    "scmd(0x%p), outstanding for %u ms & timeout %u ms\n",
	    scmd, jiffies_to_msecs(jiffies - scmd->jiffies_at_alloc),
	    (scsi_cmd_to_rq(scmd)->timeout / HZ) * 1000);
	_scsih_tm_display_info(ioc, scmd);

[vonericsen]

Hi @Forza-tng,

Thanks for reporting this.
What is odd is that this is happening on the first download segment of many segments that will be transmitted....that is bizarre since these generally complete quickly (milliseconds at most) until the activation event at the end of the download.

Can you try these to see if either of them work properly?
openSeaChest_Firmware -d <handle> --downloadFW file.lod --downloadMode segmented
openSeaChest_Firmware -d <handle> --downloadFW file.lod --downloadMode segmented --forceSCSI

[Forza-tng]

Thanks for the quick reply. I did indeed try segmented download but forgot to mention it. I will try forceSCSI when I get back to office.

[Forza-tng]

@vonericsen success with --forceSCSI

❯ /opt/openSeaChest/openSeaChest-v25.05.1-linux-i686-portable/openSeaChest_Firmware -d /dev/sg2  --downloadFW EvansBPExosX18SATA-STD-512E-SN06.LOD --downloadMode segmented --forceSCSI
==========================================================================================
 openSeaChest_Firmware - openSeaChest drive utilities - NVMe Enabled
 Copyright (c) 2014-2025 Seagate Technology LLC and/or its Affiliates, All Rights Reserved
 openSeaChest_Firmware Version: 4.3.0 X86
 Build Date: Jun  5 2025
 Today: 20250610T101719 User: root
==========================================================================================
        Forcing SCSI Drive

/dev/sg2 - ST18000NM000J-2TV103 - xxxx - SN02 - SCSI
......
Firmware Download successful
Firmware Download time (s): 5.90
Average time/segment  (ms): 71.92
Activate Time          (s): 5.80
New firmware version is SN06

[vonericsen]

@Forza-tng,

Thanks for reporting back that that worked.
I do not know why the original pass-through based method was stalling like that, but I'm glad this method worked.

I will see if I can repeat this internally and capture more information....it may be a bug in the HBA firmware that caused it, but it is difficult to say.

By forcing the SCSI commands that allows the firmware in the HBA to translate to the appropriate SATA command and do the update.
openSeaChest generally uses passthrough commands since we have had better luck with them historically.
I may have to investigate other solutions like retries or some detection of what the HBA supports to know when to switch between the two methods.

[Forza-tng]

I appreciate the help. Let me know if you want additional testing on my side. I am glad the workaround with --forceSCSI worked, so from my side this enough to close this ticket.

[vonericsen]

If I think of anything else I will let you know, but for now I want to leave this open.
I know we have some of these HBA's around inside Seagate so I just need to get some more testing to see if I can repeat the issue.

Would you be able to share the output of openSeaChest_Firmware -d <driveHandle> --llInfo?
This gets the low-level info that is read from parts of /sys to understand the HBA ID, driver version, etc which could be helpful.
Another couple pieces of info that would be helpful are openSeaChest_Firmware -d <driveHandle> --fwdlInfo and openSeaChest_Firmware -d <driveHandle> --fwdlInfo --forceSCSI -v4.
This collects capabilities for updating firmware that the drive supports. The second with --forceSCSI forces it to use the SCSI translated data from the HBA. What I'm looking for is if the HBA supports one of the newer methods available from the SAT specification because that would make it easy to identify when to use the translated method and that would be easy to auto-detect. The -v4 is so I can look at all the raw data reported in case the parsed output misses something.

The only other info that could help for debugging would be the firmware version of the HBA. I think this would have to be read by Broadcom's storecli (or similar...I cannot remember the name off the top of my head this morning 😆 ...need more coffee). That could also help since sometimes HBA firmware versions affect this kind of behavior too.

[Forza-tng]

Here is some updated info:

> ./openSeaChest_Firmware -d /dev/sg2 --llInfo 

/dev/sg2 - ST18000NM000J-2TV103 - ZR53WG0P - SN06 - ATA

---Low Level tDevice information---
        ---Drive Info---
                media type: HDD
                drive type: ATA
                interface type: SCSI/SAS/FC/etc
                zoned type: not zoned
                ---adapter info---
                        PCI/PCIe:
                        VendorID: 1000h
                        ProductID: 00E6h
                        Revision: 0000h
                ---driver info---
                        driver name: mpt3sas
                        driver version string: 48.100.00.00
                ---ata flags---
                SCSI Version: 6
                ---Passthrough Hacks---
                        Hacks were setup from the reported adapter information
                        Passthrough type: SAT/system/none
                                ---SCSI Hacks---
                                        NRSUPOP (no report supported operation codes command)
                                        SECPROT (security protocol command is supported)
                                        MXFER = 1052160B
                                ---NVMe Hacks---
                                ---ATA Hacks---
                                        CHK (SAT check condition bit is always supported)
                                        MPTXFER = 1052160B
        ---OS Info---
                handle name: /dev/sg2
                friendly name: sg2
                minimum memory alignment: 4
                ---Linux Unique info---
                        FD is valid
                        Second Handle name: /dev/sdc
                        Second Handle friendly name: sdc
                        SG Driver Version:
                                Major: 3
                                Minor: 5
                                Revision: 36
                OS read-write recommended: false
                last recorded error: 0
                File system Info:
                        No active file systems detected

openSeaChest_Firmware -d /dev/sg2 --fwdlInfo --forceSCSI -v4 generated a large amount of info, so it is attached as a file instead.

extended-info.zip

>./openSeaChest_Firmware -d /dev/sg2 --fwdlInfo
===Download Support information===
Model Number: ST18000NM000J-2TV103
Firmware Revision: SN06
Modes Supported:
        Full
        Segmented
        Deferred (Requires power cycle or activation command to activate)
Maximum Segment Size (512B Blocks): No maximum
Minimum Segment Size (512B Blocks): No minimum
Recommended Segment Size (512B Blocks): 64
Buffer Offset required (Bytes): 0

> storcli show

CLI Version = 007.2703.0000.0000 July 03, 2023
Operating system = Linux 6.12.32-dx
Status Code = 0
Status = Success
Description = None

Number of Controllers = 1
Host Name = dxsrv04.scania.com
Operating System  = Linux 6.12.32-dx
StoreLib IT Version = 07.2703.0200.0000
StoreLib IR3 Version = 16.14-0

IT System Overview :
==================

--------------------------------------------------------------------------
Ctl Model       AdapterType   VendId DevId SubVendId SubDevId PCI Address
--------------------------------------------------------------------------
  0 HBA 9500-8i   SAS3808(A0) 0x1000  0xE6    0x1000   0x4080 00:01:00:00
--------------------------------------------------------------------------

> storcli show /cALL 
Basics :
======
Controller = 0
Adapter Type =   SAS3808(A0)
Model = HBA 9500-8i
Serial Number = SKB3267949
Current System Date/time = 06/11/2025 19:52:07
Concurrent commands supported = 4992
SAS Address =  500062b20828c940
PCI Address = 00:01:00:00

Version :
=======
Firmware Package Build = 28.00.00.00
Firmware Version = 28.00.00.00
Bios Version = 09.55.00.00_28.00.00.00
NVDATA Version = 28.01.00.12
Driver Name = mpt3sas
Driver Version = 48.100.00.00

PCI Version :
===========
Vendor Id = 0x1000
Device Id = 0xE6
SubVendor Id = 0x1000
SubDevice Id = 0x4080
Host Interface = PCIE
Device Interface = SAS-12G
Bus Number = 1
Device Number = 0
Function Number = 0
Domain ID = 0

Pending Images in Flash :
=======================
Image name = No pending images


Status :
======
Controller Status = OK
Memory Correctable Errors = 0
Memory Uncorrectable Errors = 0
Bios was not detected during boot = No
Controller has booted into safe mode = No
Controller has booted into certificate provision mode = No
Package Stamp Mismatch = No


Supported Adapter Operations :
============================
Alarm Control = No
Cluster Support = No
Self Diagnostic = No
Deny SCSI Passthrough = No
Deny SMP Passthrough = No
Deny STP Passthrough = No
Support more than 8 Phys = Yes
FW and Event Time in GMT = No
Support Enclosure Enumeration = Yes
Support Allowed Operations = Yes
Support Multipath = Yes
Support Security = Yes
Support Config Page Model = No
Support the OCE without adding drives = No
support EKM = No
Snapshot Enabled = No
Support PFK = No
Support PI = No
Support Shield State = No
Support Set Link Speed = No
Support JBOD = No
Disable Online PFK Change = No
Real Time Scheduler = No
Support Reset Now = No
Support Emulated Drives = No
Support Secure Boot = Yes
Support Platform Security = No
Support Package Stamp Mismatch Reporting = Yes
Support PSOC Update = No
Support PSOC Part Information = No
Support PSOC Version Information = No

Policies :
========

Policies Table :
==============

------------------------------------------------
Policy                          Current Default
------------------------------------------------
Predictive Fail Poll Interval   0 sec
Interrupt Throttle Active Count 0
Interrupt Throttle Completion   0 us
Rebuild Rate                    0 %     30%
PR Rate                         0 %     30%
BGI Rate                        0 %     30%
Check Consistency Rate          0 %     30%
Reconstruction Rate             0 %     30%
Cache Flush Interval            0s
------------------------------------------------

Flush Time(Default) = 4s
Drive Coercion Mode = none
Auto Rebuild = Off
Battery Warning = Off
ECC Bucket Size = 0
ECC Bucket Leak Rate (hrs) = 0
Restore HotSpare on Insertion = Off
Expose Enclosure Devices = Off
Maintain PD Fail History = Off
Reorder Host Requests = On
Auto detect BackPlane = SGPIO/i2c SEP
Load Balance Mode = None
Security Key Assigned = Off
Disable Online Controller Reset = Off
Use drive activity for locate = Off


Boot :
====
Max Drives to Spinup at One Time = 2
Maximum number of direct attached drives to spin up in 1 min = 60
Delay Among Spinup Groups (sec) = 2
Allow Boot with Preserved Cache = On

Defaults :
========
Phy Polarity = 0
Phy PolaritySplit = 0
Cached IO = Off
Default spin down time (mins) = 0
Coercion Mode = None
ZCR Config = Unknown
Max Chained Enclosures = 0
Direct PD Mapping = No
Restore Hot Spare on Insertion = No
Expose Enclosure Devices = No
Maintain PD Fail History = No
Zero Based Enclosure Enumeration = No
Disable Puncturing = No
Un-Certified Hard Disk Drives = Block
SMART Mode = Mode 6
Enable LED Header = No
LED Show Drive Activity = No
Dirty LED Shows Drive Activity = No
EnableCrashDump = No
Disable Online Controller Reset = No
Treat Single span R1E as R10 = No
Power Saving option = Enable
TTY Log In Flash = No
Auto Enhanced Import = No
Enable Shield State = No
Time taken to detect CME = 60 sec


Capabilities :
============
Supported Drives = SAS, SATA, NVMe
Enable JBOD = Yes
Max Parallel Commands = 4992
Max SGE Count = 128
Max Data Transfer Size = 32 sectors
Max Strips PerIO = 0
Max Configurable CacheCade Size = 0
Min Strip Size = 512Bytes
Max Strip Size = 512Bytes

Scheduled Tasks = NA

Secure Boot :
===========
Secure Boot Enabled = Yes
Controller in Soft Secure Mode = No
Controller in Hard Secure Mode = Yes
Key Update Pending = No
Remaining Secure Boot Key Slots = 7


Security Protocol properties :
============================
Security Protocol = None


Enclosure Information :
=====================

---------------------------------------------------------------
EID State Slots PD PS Fans TSs Alms SIM ProdID  VendorSpecific
---------------------------------------------------------------
 19 OK       24 19  0    0   2    0   0 SC846-P x40-66.16.11.0
 31 OK       12 12  0    0   2    0   0 SC826-P x28-66.16.11.0
---------------------------------------------------------------

[vonericsen]

@Forza-tng,
Thank you for sharing all that info!
That will be very helpful for debugging and repeating this issue!

I realized one other thing that will be helpful to have:
openSeaChest_Firmware -d <handle> -i --SATInfo -v4
This reports the data we can get from both the translator (HBA) and the SATA drive so I can compare for other differences and capabilities.
In this case I want to see the very verbose output to check all the raw bytes to make sure there isn't anything else we can use in deciding when to attempt the translated download versus the passthrough download.

[Forza-tng]

@Forza-tng, Thank you for sharing all that info! That will be very helpful for debugging and repeating this issue!

I realized one other thing that will be helpful to have: openSeaChest_Firmware -d <handle> -i --SATInfo -v4 This reports the data we can get from both the translator (HBA) and the SATA drive so I can compare for other differences and capabilities. In this case I want to see the very verbose output to check all the raw bytes to make sure there isn't anything else we can use in deciding when to attempt the translated download versus the passthrough download.

I will check this on Friday as I am not in office tomorrow.

Update: @vonericsen here is the new verbose info:
verboseinfo_firmware_SATInfo.zip

[vonericsen]

I now know the root cause of this and it was fixed in Seagate/opensea-transport#23

The issue that I missed in my first viewing of the verbose log is that the download command issues was the PIO mode version, however it was submitted with the protocol set to UDMA which caused the hang.

[vonericsen]

Closing since this has been fixed in v25.05.2 release