Installing and configuring IBM Fault Analyzer for z/OS Abend Intelligence

About this task

Deploy the on-host IBM Fault Analyzer for z/OS Abend Intelligence pipeline to z/OS UNIX System Services (USS) to enable real-time AI analysis of application abends.

For general information about this enhancement, see The AI analysis report.

To install IBM Fault Analyzer for z/OS Abend Intelligence:

Procedure

  1. Download Python 3.10 from https://www.python.org/downloads/ and deploy it and the required packages to your z/OS UNIX System Services (USS) environment. The required packages are found in data set IDI.SIDIDOC1 member IDIZREQ. The Python engine imports components from the schemas and custom_exceptions packages. You must recreate this mandatory package subdirectory structure on your USS environment. If copied to a flat directory, the startup fails. 
    src/
├── IDIZMAIN.py            # entry point
├── IDIZUTIL.py
├── IDIZCNST.py
├── IDIZCONF.py
├── IDIZREQ.txt            # dependency pins
├── schemas/
│   ├── IDIZFDIN.py
│   ├── IDIZFDEX.py
│   └── IDIZFILE.py
└── custom_exceptions/
    └── IDIZEXCP.py
    Important:
    The pyzfile required package only works on Python 3.10. If using Python 3.11, IBM Z Open Automation Utilities (ZOAU) is mandatory.
    1. Verify the log directory path in IDIZCNST.py.

      Open IDIZCNST.py in an editor and check the LOG_DIRECTORY setting:

      LOG_DIRECTORY = "/tmp"
      Important:

      Ensure that LOG_DIRECTORY specifies a z/OS UNIX directory where background batch job user IDs have write access (for example, /tmp or a dedicated directory such as /var/fai/logs). If the user ID lacks write permission to this directory, log file creation fails and no diagnostic execution logs are generated.

  2. Create an FB 80 data set containing your LLM credentials and environment definitions. For details, see The STDENV environment variables for Abend Intelligence.
  3. Register the IBM Fault Analyzer for z/OS Abend Intelligence user exits via the IDIOPTS DDname in your abending test job. 
    DATASETS(IDIEXEC(IDI.SIDISAM1))
EXITS(
    NOTIFY(REXX(IDISUEAN)) 
    FORMAT(REXX(IDISSHOW))
)
    The IDISUEAN and IDISSHOW user exits are provided in the IDI.SIDISAM1 data set. Ensure these exits are available to Fault Analyzer via the IDIEXEC DDname.
    IDISUEAN
    This is a Fault Analyzer Notification user exit written in REXX that captures environment variables from the active fault recording session, invokes the orchestration launcher (IDISBFCR) to build a JSON payload in /tmp/, and submits an asynchronous job to the internal reader to analyze the failure using a large language model (LLM).
    Verify or change the AI Service Configuration definitions in this exit as appropriate for your installation:
    Table 1. IDISUEAN REXX Exit Configuration Variables
    Variable Name Example Value Description
    LLM_PROTOCOL_TO_USE 'watsonx' or 'ollama' Specifies the target large language model backend protocol: LLAMA or WATSONX.
    LLM_SERVICE_ENDPOINT 'http://13.82.3.70:11434' Specifies the base URL of the LLM endpoint (required for Ollama).
    LLM_MODEL_TO_USE 'llama3:latest' or 'ibm/granite-3-8b-instruct' Specifies the model identifier to invoke on the AI backend.
    USE_SOURCE_CODE YES Attempts DWARF/LANGX/LISTING source extraction.
    PROGRAM_MODE BF Only BF is supported.
    PYTHON_EXEC_TO_USE '/usr/lpp/IBM/cyp/v3r10/pyz/bin/python3' Specifies the fully qualified USS path to the Python 3.10 executable.
    PYTHON_MAIN_TO_USE '/usr/local/bin/fai/src/IDIZMAIN.py' Specifies the fully qualified USS path to the IDIZMAIN.py entry point.
    STDENV_TO_USE USERID()"FAI.ENVS(LOCALENV)" Specifies the RACF-protected FB 80 data set containing API credentials shown in step 1.
    IDIOPTS_TO_USE USERID()".FAI.OPTIONS(FAIOPTS)" IBM Fault Analyzer for z/OS Abend Intelligence options data set used in the generated JCL. No specific options are required to enable IBM Fault Analyzer for z/OS Abend Intelligence.
    USE_ZOAU_MODE NO Setting this variable to YES passes USE_ZOAU=YES into the JSON payload.
    IDISSHOW
    This is a Fault Analyzer Formatting user exit written in REXX that reads the Markdown report data set <userid>.FAI.REPORT(<fault_id>) and converts Markdown elements to Fault Analyzer Formatting user exit tags (see Formatting tags).
  4. Submit the abending test job with a sample JSON payload in /tmp/ to verify outbound TCP/IP connectivity to your LLM endpoint.
  5. Verify the installation. You can verify the installation in the following ways:
    1. JES Spool Check: Open STDOUT from the automatically submitted IBM Fault Analyzer for z/OS Abend Intelligence BPXBATCH job in SDSF and verify that the step finishes with RC=0 and displays the ### FAI Job Summary ### block with Status: Success.
      Note:
      A return code of RC=253 indicates that the pipeline executed successfully, but the LLM endpoint timed out or was unreachable over TCP/IP. Verify TCP/IP firewall rules and the LLM_SERVICE_ENDPOINT setting in IDISUEAN.
    2. AI Report Data Set Member Check: View data set <userid>.FAI.REPORT(<fault_id>), where <fault_id> is the fault ID created by the test job in step 4, using for example ISPF 3.4 to verify that the Markdown report content was written.
    3. Fault Analyzer Report Check: Perform reanalysis of the fault entry written in step 4 and confirm that the AI report is included in the User section of the report.
  6. If the testing in step 5 was successful, propagate the IDIOPTS DDname options in step 3 to the IDICNFxx parmlib member to make IBM Fault Analyzer for z/OS Abend Intelligence available to all jobs and users.