Test Case Execution and Reporting

Execution of test cases is possible in several ways

• directly from the Int4 APITester Cockpit

• by using available API’s: REST, SOAP, RFC, SAP GUI Transaction, read here for more details: API to integrate Int4 Suite with other software

• by running SAP GUI dedicated transaction /INT4/IFTT_RUN - this also allows scheduling test runs in the background, read more about background scheduling here: Background processing

Triggering test execution from Int4 APITester Cockpit

Start with selecting a single folder or a set of folders containing the test cases or test scenarios to be executed. List of relevant test cases will open on the right hand side of the screen.

Open

From this moment, all test cases can be executed by clicking the “Execute All” button on the screen.

It is possible to select specific test cases, using the checkbox on the left of the test cases. In such case, the “Execute All” button changes to “Execute Selected”

Execution with parameters

Execution of test cases is controlled by specific settings. If they are not explicitly selected, defaults are used. To control test execution settings, use the down arrow beside the “Execute” button and choose to “Execute w/Parameters”

Open

Execute w/Parameters button in the dropdown

Once selected, the Execution Parameters window pops up.

Basic parameters

• Test Run Name - name to identify test run - free form text description, defaults to date and time

• Environment - the environment where the test cases should be executed, defaults to the one specified on test case, or on folder level

• Test Run type - select the applicable type of test run - used for internal reporting to differentiate between various test runs (e.g. unit testing, regression, test design)

• Execute New TCs Only - will execute only these test scripts which were not yet tested

• Debug Mode - provides a detailed log, useful for test design and debugging

• Execute via SAP GUI - downloads and runs SAP GUI link, which (on configured system) will start SAP GUI and open the testing transaction. This is needed for certain scenarios which utilize SAP GUI based eCATT Scripts.

• Number of Attempts to Read TC Results - the number of times that Int4 Suite will poll for data after test execution

• Delay Between Attempts to Read TC Result - delay between each consecutive attempt to read test results

• Delay Between Execution/Validation

  • For Each Test Case - delay between test case execution start and first attempt to read TC results, for each test case

  • Once Per Test Run - as above, but applied to all test cases in parallel.

Additional Parameters

For certain scenarios, You might want to override the parameters defined in the Automation Object for a particular test run. Choose the parameter name and provide the new value. Multiple parameters can be altered.

Load / Performance Testing

Int4 Suite can be used to test the behaviour of integration platform under load. This is enabled by following features

• Repeat N Times - test cases run will be repeated as many times, as specified

• Skip Assertions - this will skip the output validation step in test case execution

• Execute Using Parallel Processing - normally Int4 starts all test cases in sequence and then captures the output in parallel processing, checking this box will trigger all the tests (excluding the ones which refer a parent) in parallel

Starting test

Clicking “Execute” or confirming the parameters with “OK” will trigger a test run. A new tab opens with Test Run Details

Execution Report - Test Run Summary

The test run summary screen details the specifics of the test run and shows the status, for each test case in the list at the bottom and summary test status on the top.

In the summary section, user can navigate to:

• Performance Analysis ( if performance data is available )

• Error Summary

• Legacy Report

Test Case information follows the details from the APITester Cockpit. Clicking on the row opens the test details for particular test case.

Error Summary

In this section errors from all of the test cases are summed up, aggregated by text and location, showing message count. User can see the list of test cases that include specific message by clicking on a button at the end of the row. Clicking on one of the test cases will navigate to its run details.

Execution Report - Test Case Run Details

The Basic Data section contains general details about the test case executed and the execution context.

Execution Information contains the log of the test execution. In normal mode the log is brief and displays key information to identify the test artifacts (e.g. the message ID’s on the integration platform). Enabling Debug Info in Execution Parameters will provide more technical details on test execution and issues.

The Variables section contains the list of Variables from the relevant Automation Object with description and values.

• Reference value - the value at the beginning of test case execution, usually mapping to the value read out from the test case Payload (from reference document)

• Current value - the value at the end of test execution

The Input Test Message section occurs for Inbound test types, and references in details the message that was injected into the integration platform

Results Comparison

This is by far the most complex section of the report. It is split in two sections

• Payload comparison after processing - this section compares the output of the integration platform to the reference document from the Test Case Payload

• SAP Backend Document Validation - if the Automation Object specifies a DB comparison object, Int4 APITester will present the details of comparing the reference document with the once captured in current test execution

Payloads after processing

Payload Comparison shows the reference message (on the left) and current execution (on the right) in a structurally aligned way, highlighting the differences.

• Yellow - difference is acceptable and triggers a warning, the test case will pass

• Red - difference is unexpected and triggers an error, the test case will fail

The arrows on the left allow to quickly jump to next difference

Black arrows jump to next difference

Red arrows jump to next error

SAP Backend Document Validation

Backend validation feature connects to the backend databases, extract the relevant documents and compares them accordingly to DB Validation Ruleset assigned to the Automation Object.

Depending on Automation Object settings, backend document can be fetched during test case creation or test case execution.

The comparison is executed on field level, for each of the defined tables and fields. Similarly to the Payload Validation, the expectation is that documents will be identical. Known and expected differences can be defined in the DB Validation Ruleset for fine control of the results.

By default, only current execution data is shown. Additional checkboxes are available to control report display

• Show reference document - shows the reference document data in gray

• Show technical field names - shows the technical names of tables and fields

Color coding is used to highlight report results

• Green - data is the same or matching, based on DB Validation rules

• Yellow - difference is acceptable and triggers a warning, the test case will pass

• Red - difference is unexpected and triggers an error, the test case will fail

The arrows on the first left column allow jumping between errors quickly. Thanks to them, you can jump into the next/previous row with differences

Rerunning and revalidating test cases

To rerun or revalidate test cases use Test Runs Report