The LeCroy SASTracker Command Analyzer monitors command execution and completion in SAS and SATA storage subsystems. By monitoring all commands issued within a matrix that can involve up to 8 individual initiators and 128 independent targets, the SASTracker Analyzer enables system integrators to identify commands that hang and that are slow to complete. The system is capable of monitoring over 10,000 pending commands and maintaining a log file of over 8 million commands.
By focusing its analysis on command execution, command timeout and quality-of-service problems, the SASTracker Analyzer helps server and storage OEMs optimize system performance and develop more reliable storage subsystems.
Like a protocol analyzer, SASTracker sits in the data path on a live system; but it does not record bit-level details. Instead, it monitors all commands issued and tracks their completion, reporting details such as initiator ID, target ID, op code and metrics such as command execution time and queue depth. The user may set a timeout trigger and the system will monitor and record all pending commands until any command exceeds the user-defined timeout, at which point the system will trigger and report the condition to the user.
Due to increasing bus traffic and the large number of pending commands that may exist concurrently, errors occur in storage subsystems that only become apparent as the complexity of the system increases. During the process of system integration, the user can initially set the timeout trigger to a large value to identify commands that fail to complete and focus on the associated hardware, programming or protocol errors. As these issues are identified and resolved, the user can reduce the timeout trigger and start to focus on delayed and data-starved command conditions. This process systematically identifies problems and inefficiencies and allows users to optimize performance in the storage subsystem.
The hidden cost of command latency
Advanced SAS and SATA storage subsystems make extensive use of command queuing to improve performance. Command queuing allows each target drive (typically a disk drive) to accept multiple commands into a “queue” and then reorder the commands to allow efficient completion of all commands. For example, if a drive head was positioned at sector A and was to pass through sectors B, C and D as the disk spins, commands received for data from sectors D, B, C and A would be reorganized and executed in the order A, B, C and D. The result is faster completion of the total tasks required.
The example above is very simple and uses a queue depth of only 4 commands. The queue depth defined by the SCSI specification allows a 16-bit queue tag, or a queue depth of over 60,000 commands. In practice, most drives limit queue depth to 128 commands, but the complexity of the resulting command execution, especially in a system that involves multiple drives, is apparent.
The result is that systems that work in simple environments often encounter problems as the system complexity increases. Commands may be issued and never completed, or may be delayed substantially, resulting in the rapid transmission of many commands but poor overall system performance. Protocols attempt to address these issues by providing higher level command timeouts and establishing priorities for commands, but tracking these issues in a typical storage subsystem can become a task far beyond the reach of protocol analyzers and similar tools.
The SASTracker Command Analyzer was designed to address precisely this problem. Either use alone or in conjunction with a protocol analyzer, the SASTracker Analyzer allows the user to identify and systematically eliminate performance problems, resulting in a storage subsystem with optimum performance.
With new SAS and SATA based subsystems reaching the market, developers are learning first hand how difficult it is to isolate tagged command queuing problems across large subsystems. Random I/O operations (commonly associated with transaction processing) are more likely to stress device queue algorithms during periods of peak activity. It’s here that subsystems that work flawlessly at low link utilization rates may exhibit problems with stranded commands during periods of high disk activity.
Multi-initiator environments found in clustering applications create added complexity because each SAS controller can transmit queued commands to each device in the subsystem concurrently. Assuming a blade server with 128 tags (typical SAS HDD queue depth) x 128 HDDs = over 16,000 potential outstanding queued commands. To effectively track every outstanding command to completion requires an analyzer capable of maintaining and monitoring timing on thousands of queued operations.
Commands that are ACKed by a device but fail to complete or are slow to complete are surprisingly difficult to isolate. SAS and SATA based disk drives configured in RAID environments aggravate the problem because they can generate data rates up 12 Gbps. Even with maximum filtering techniques, these high sustained data rates drastically reduce the amount of elapsed time that can be recorded with conventional analyzers.
Usage Cases
The SASTracker Command Analyzer is a new type of analysis tool specifically designed to debug intermittent command latency problems. It’s the first system capable of maintaining timing context and isolating command latency issues across complex transaction processing storage environments. It tracks every command issued with an independent timer and triggers in real time when it detects slow or incomplete commands. The SASTracker Analyzer displays a log of all commands with a marker at any timeout violation. This alone can save weeks of manual analysis by identifying the exact sequence of events that preceded the timeout condition.
The SASTracker system transparently taps from 1 to 4 SAS or SATA links. It includes a large real-time recording memory allowing it to track up to 8 initiators and 128 unique targets in a single session. This provides concurrent, end-to-end monitoring of over 16,000 pending I_T_L_Queue commands. Multiple Tracker systems can be used at multiple points in the storage fabric to ensure QOS at the application level.
The SASTracker system’s adjustable command threshold timer (from 50 us to 10 seconds) allows test engineers to identify a range of issues?from fatal errors to marginal devices. Conventional protocol analyzers are incapable of maintaining timing context for thousands of pending commands.
Quality of Service (QOS) Analysis – Blade servers and “grid” computing applications are ideally suited for SAS-based storage architectures. In this environment, server utilization can be dependent on the host controller’s ability to efficiently distribute IO operations. From the storage perspective, this can include multi-initiator environment where multiple servers are vying for access to the same storage resources. Ensuring that QOS is maintained at the application layer requires monitoring I/O operations for extended periods.
Continuous Computing - SAS supports fail-over to deliver fault tolerance for mission critical applications. Dual ported disk drives are used to provide an alternate pathway to a device if something happens to the primary path. Monitoring queue depth during failover / recovery is another area that can benefit from the SASTracker Analyzer’s specialized recording. When an initiator stops responding, it is vital that the storage devices seamlessly switch to the new host controller with minimal impact on command completion times. Monitoring queue depth during simulated fail over is key application for the SASTracker Analyzer.
The log displays each operation with the Initiator, Target ID, and other command level parameters including LBA, OP Code and transfer length. Each entry includes an absolute timestamp and delta between Command and Status. It’s capable of maintaining a history of the last 8 million commands with markers at each time-out violation.
Economical and Upgradeable
The SASTracker Analyzer option is built on the same platform as LeCroy’s market leading protocol analyzer for SAS and SATA. The Tracker option can be added to an existing protocol analyzer to allow a unique multi-purpose test capability. LeCroy SASTracer analyzer can be re-configured into a SASTracker in the field as needed by users. Reconfiguring the system allows users in the field to switch between the “bit-level” protocol analyzer and the “command” level Tracker capability.
LeCroy’s SAS analysis platform now provides the flexibility to evolve with the product development lifecycle. Equipment originally purchased for R&D test can be re-deployed in the field to help isolate marginal devices or software issues. But the real advantage of choosing LeCroy’s verification platform is its ability to shorten the time-to-market for SAS and SATA based storage solutions.
