1. lxMicMonitor


Copyright (c) 2013 Intel Corporation. All rights reserved.

Version_number=2.01.1.0001

1.1. Overview

Copyright (c) 2013 Intel Corporation. All rights reserved.

Version_number:  2.01.1.0001
Released:        25-April-2013

Purpose
This module is used to monitor the host kernel message log, during testing of Intel Many Integrated Cores (MIC) PCI-e 
cards. The test depends on the MIC SW Stack which consists of:
*	MIC Board Tools (MICSYSSW)
*	MIC Host Driver
*	MIC uOS
These components are packaged in the RPM file:
intel-mic-knc-kmod-<version>.el6.x86_64.rpm
Where <version> should be at least: 2.1.3126-13...
Earlier versions of this RPM file will not be compatible with this release. 

This module's Check MCE Log, subtest uses the MIC tools to boot the MIC card's uOS and check the KNC (Knight's 
Corner), EEPROM log, to see if it empty. The Montor Log File, subtest does not interact with the KNC.  It monitors a 
host side log file, /var/log/messages, by default, and reports any interesting MIC related error messages.

Runtime Environment
The Intel(R) Modular Test Architecture Test Executive for Linux template module.  The MIC SW Stack must be installed 
on the Linux operating system.

License Agreement
All code is under the Intel(R) software License which can be found at http://legal.intel.com. A copy of this agreement 
must be included if code   is released outside Intel.

Hardware Description
This test requires the presence of a MIC PCI-e card in the host system.





1.2. Version Information

 
Date		Version			Owner		Comments
04/25/2013	2.01.1.0001		DKJ		First released version 	

		

1.3. Installation

Files needed to run this module:
	
File	Description
lxMicMonitor		Executable image.
lxMicMonitor.snx	Configuration file.
lxMicMonitor.txt	Text only help file.
lxMicMonitor.chm	Compressed HTML help file.
lxmicstart			Script to boot MIC card. Provided by lxMicBoot

NOTE: This module is currently set to work with only one MIC card in a host system.

lxMicBoot:
This package depends on the lxMicBoot test package to provide the lxmicstart script which are use to boot the MIC uOS.  
The lxMcBoot version required to support this version of lxMicCfg, must be at least 2.13.1.0001

MPSS RPM: 
A supported version of the MPSS RPM must be installed on the host system. Both of the following RPMs must be 
installed (replace version with the correct version);
intel-mic-version.el6_x86_64.rpm
intel-mic-knc-kmod-version.el6.x86_64.rpm

For Example:
intel-mic-2.1.1818-1-2.6.32-71.el6.x86_64.rpm
intel-mic-knc-kmod-2.1.1818-1-2.6.32-71.el6.x86_64.rpm

Please see project contact for copies of these RPMs and any updates on supported RPM versions. 
RPMs may be installed on the host system with this command: rpm -ihv rpm_file_name.rpm

Host board: 	The host board must be supported by the MPSS RPM team. Please see project contact for a list 
of supported boards. This release was tested with Intel(r) Workstation Board W2600CR2.

Operating Systems:	The OS on the host board must be supported by the MPSS team. Currently, the supported OS 
are Red Hat Enterprise Linux 6.0 and Red Hat Desktop 6.0.

SELinux: 	There is a known issue with interaction between the MPSS SW and SELinux, that makes MIC 
uOS boot unreliable. Because of this SELinux should be disabled when running tests which 
require MIC uOS boots and reboots, such as the tests in this test module.  Note:  CTC normally 
puts SELinux, if it is enabled, in permissive mode when CTC is running. However this is not 
solve the problem with SELinux interfering with MIC uOS boot.  SELinux must be disabled to 
avoid this problem.

To Disable SELinux:
*	On the host system, as root, edit /etc/selinux/config
*	Change the line which starts SELINUX=...  to SELINUX=disabled
*	Reboot the system
*	After the reboot use the command: sestatus to verify that 	SELinux is disabled.

To Re-Enable SELinux:
*	On the host system, as root, edit /etc/selinux/config
*	Change the line: SELINUX=disabled to SELINUX=permissive
*	Reboot the system.  During this reboot SELinux will detect the 	configuration change and will 
	force file label update, and then do a second reboot.
*	After reboot use the command: fixfiles relabel to update labels 	on all the files monitored by 
	SELinux. Responding N to question about cleaning out /tmp is OK.
*	Edit /etc/selinux/config and change SELINUX=permissive to 	SELINUX=enabled
*	Then reboot again
*	After the reboot use the command: sestatus to verify that 	SELinux is enabled.  If the system 
	automatically starts CTC on reboot, sestatus will show the 	SELinux status as permissive since 
	CTC automatically sets SELinux to permissive mode when it 	starts, but only if SELinux is 
	enabled.

MIC card:	A supported MIC card must be plugged into a PCI-E slot on the supported host board. Please see 
project contact for a list of supported boards.

lxMicMonitor:	Copy the test module files, listed above, to the CTC directory.






1.4. Sub-Tests


Test 0	Auto-Configuration

Test 1	Check MCE Log

Test 2	Monitor Log File



1.5. Test 0 - Auto-Configuration

Overview

This subtest performs any required setup for the test executive. Currently, there is no required setup for this test 
module.

Parameter(s)
None

Test Execution Time
 Less than one second	

Dependencies/Limitations
None





1.6. Test 1 - Check MCE Log

Overview

This subtest checks the size of the EEPROM log on the MIC/KNC board.  The log is read, by booting the uOS on the 
MIC, and reading mic0:/proc/elog 
If this EEPROM log contains more than MaxElogSize bytes, the test will fail.
If the test fails, and ElogSaveFileName is set, the contents of mic0:/proc/elog will be saved in a filename created by reading the contents of SerialNumFileName then  appending a timestamp, in the format: YYYYMMDDHHMMSS, followed by  ElogSaveFileName.  So if, for example, SerialNumFileName contains ADKC25101824G65757521 and ElogSaveFileName is set to .elog the name of the name used to save the contents of mic0:/proc/elog will be: ADKC25101824G65757521_20130502141637.elog

Parameter(s) 
MaxElogSize
ElogSaveFileName
SerialNumFileName

Test Execution Time
Varies, but basically the time it takes to boot the uOS on the MIC PCI-e card.

Dependencies/Limitations
None.





1.7. Test 2 - Monitor Log File

Overview

When started this test module opens MessageFileName and scans to the end of this file.  Then it waits 
MonitorTimeSec while it reads any new messages written to that file and checks for MIC related messages, which are 
known to be associated with failures.  If one of these MIC messages is detected, the test will report and error, except in 
cases where the PostCodeArray specifies a different action.

Known MIC related messages, that this module checks for:

*	micras: These messages are generated by the MIC RAS handler and ether report a MIC Machine Check
	Error (MCE), or an action taken as a resutl of one of these. If a the message indicates that it
	is an uncorrectable MCE which has been logged to the MIC's EEPROM log, then an error is reported.
	If the message is for a correctable error that was not logged to the MIC's EEPROM log, then an
	Advisory is reported.

*	RESETFAIL: Any message containing this keyword will cause a CTC error message to be logged.

*	MIC BOOT: A Waiting for MIC 0 boot ... message which has a wait time greater than MicMaxBootTimeSec
	will cause a CTC error message to be logged.
*	MIC Post Code: Any mic0: Resetting (Post Code ... message which contains one of the post codes in
	PostCodeArray will cause the Action in PostCodeArray for that post code to be taken.

If any CTC error messages are logged the test fails.
If the test does not report any CTC error messages, during the MonitorTimeSec seconds, it monitors MessageFileName 
the test passes.
 
Parameter(s) 
MessageFileName
MonitorTimeSec
PostCodeArray
MicMaxBootTimeSec

Test Execution Time
The test will execute for MonitorTimeSec seconds.

Dependencies/Limitations
None.





1.8. Parameters


ElogCheckTimeSec

ElogSaveFileName

MaxElogSize

SerialNumFileName

MessageFileName

MonitorTimeSec

MicMaxBootTimeSec

PostCodeArray



1.9. ElogCheckTimeSec

Description
Number of seconds the Check MCE Log test will wait for the MIC uOS to reply with the contents of /proc/elog, before 
timing out.

Default
120

Dependencies
Required

Used by
Test 1 - Check MCE Log


See Also
Parameters

1.10. ElogSaveFileName

Description
Base name of file, on host system, where Test 1 - Check MCE Log should save contents of MIC card's /proc/elog.

Default
.elog

Dependencies
Optional

Used by
Test 1 - Check MCE Log

See Also
Parameters

1.11. MaxElogSize

Description
Maximum allowable size of /proc/elog on MIC card. 

Default
0

Dependencies
Required

Used by
Test 1 - Check MCE Log


See Also
Parameters

1.12. SerialNumFileName

Description
Name of file containing the MIC card's unique identification number (i.e. Serial Number). 

Default
KNC_SN.txt

Dependencies
Optional

Used by
Test 1 - Check MCE Log


See Also
Parameters

1.13. MessageFileName

Description
File path of the message file which Test 2 - Monitor Log File monitors for interesting MIC related messages.

Default
/var/log/messages

Dependencies
Required

Used by
Test 2 - Monitor Log File


See Also
Parameters

1.14. MonitorTimeSec

Description
Number of seconds, that Test 2 - Monitor Log File will spend monitoring MessageFileName.

Default
600

Dependencies
Required

Used by
Test 2 - Monitor Log File


See Also
Parameters

1.15. MicMaxBootTimeSec

Description
When Test 2 - Monitor Log File detects a MIC waiting for uOS boot message, this is the maximum number of seconds 
the log message can say it has been waiting before an error is reported.

Default
100

Dependencies
Required

Used by
Test 2 - Monitor Log File


See Also
Parameters

1.16. PostCodeArray

Description
Parameter array, where each row contains a MIC Post Code value and the Action that should be taken when that Post 
Code is found in MessageFileName.
Supported actions are:
*	Error	Log a CTC Error message when the MIC Post Code is detected
*	Advisory	Log a CTC Advisory message when the MIC Post Code is detected
*	Info		Log a CTC Verbose message when teh MIC Post Code is detected

Default

Index
MIC Post Code
Action
0
F2
Error
1
6P
Error
2
17
Error

Dependencies
Required

Used by
Test 2 - Monitor Log File


See Also
Parameters

1.17. Error Codes


ERROR codes formatted as 0xMMMTTEEE 

MMM - Module number  
TT      - Test number 
EEE   - Error code number, as below:

EEE	Message
001  	Invalid test number ___
002  	Error setting parameter ____
003  	Internal error: ___
004  	Unable to retrieve parameter: ____
005  	Message file ____, ____, errno: ____
006	MIC MCE ERROR Logged with status: ____, ____
007	MIC Reset Fail: ____
008	Unable to Open File: ____
009	Elog on MIC too big! Elog size: ____ Max size: ____
00A	uOS Boot Failed: ____
00B	Time waiting for MIC boot exceeded: ____: ____
00C	Invalid value for parameter: ____
00D	File Operation Error, file: ____, ____, errno: ____
00E	MIC Post Code ____ message found: ____


Debug Help:

Error 003 indicates that an internal error has been detected.
Error 005 indicates a problem accessing the message file: MessageFileName
Error 006 indicates a MIC MCE Uncorrectable, or correctable which exceeded allowable frequency, was
	  logged to the  MIC's EEPROM log.
Error 007 indicates a MIC RESETFAIL messages was detected.
Error 008  indicates a problem opening temporary files in the CTC directory.
Error 00A indicates a problem booting the uOS on the MIC card. Check setup to verify that MIC uOS can boot.
Error 00B indicates a waiting for MIC boot message in has been waiting more than MicMaxBootTimeSec seconds.
Error 00C The indicated parameter does not have a valid value.
Error 00D General format for reporting a file operation error. Check CTC setup and permissions,
	  to verify that the test  module can access local files.
Error 00E A MIC Post Code message, with a post code listed in PostCodeArray was found.

