Sphinx-Test-Reports

Sphinx-Test-Reports shows test results inside Sphinx documentations.

It provides the following features:

• test-file: Documents all test cases from a junit-based xml file.

• test-suite: Documents a specific test-suite and its test-cases.

• test-case: Documents a single test-case from a given file and suite.

• test-report: Creates a report from a test file, including tables and more for analysis.

• test-results: Creates a simple table of test cases inside a given file.

• test-env: Documents the used test-environment. Based on tox-envreport.

Introduction

A single documented test-case looks like this:

Test-Case: test_add_source_parser TC_F7984

doctype: .rst style: tr_passed delete: False jinja_content: False file: ../tests/doc_test/utils/pytest_sphinx_data.xml suite: pytest case: test_add_source_parser case_name: test_add_source_parser classname: None time: 0.009029150009155273 result: passed

Custom Message: The result is passed with an execution time of 0.009029150009155273. Now follows automatically generated output like system-out, messages and text of the test-case. System-out: # testroot: root # builder: html # srcdir: /tmp/add_source_parser # outdir: /tmp/add_source_parser/_build/html # status: #x1B[01mRunning Sphinx v1.7+/aecb0082#x1B[39;49;00m #x1B[01mloading pickled environment... #x1B[39;49;00mnot yet created # warning:

Take a look into our pytest example to see the complete result of all Sphinx tests (1154 test cases!).

The objects created by Sphinx-Test-Reports are based on Sphinx-Needs. So all features for filtering, sorting and showing data is supported.

As example, here is a shorten list of tests results from the Sphinx-pytest example:

ID	Title	Result
REPORT_0EA_049F6	test_insert_similar	passed
REPORT_0EA_140FB	test_rstdim_to_latexdim	passed
REPORT_0EA_57BA7	test_theme_api	failure
REPORT_0EA_734D6	test_build	passed
REPORT_0EA_73FAE	test_build_latex_doc[lualatex-howto]	failure
REPORT_0EA_8F625	test_generate	passed
REPORT_0EA_EB670	test_texinfo	skipped

Content

• Installation
• Directives
  • test-file
  • test-suite
  • test-case
  • test-report
  • test-results
  • test-env
• Configuration
  • tr_rootdir
  • tr_file
  • tr_suite
  • tr_case
  • tr_report_template
  • tr_suite_id_length
  • tr_case_id_length
• Filtering Test Data
  • Filterable data
  • needtable - A table for all tests
  • needlist - A simple list
  • needflow - Flow charts of linked test data
• Dynamic functions
  • tr_link
• Examples
  • Importing
  • Filtering
  • Test framework related examples
• Support
  • Professional Support
• Changelog
  • 1.0.2
  • 1.0.1
  • 1.0.0
  • 0.3.7
  • 0.3.6
  • 0.3.5
  • 0.3.4
  • 0.3.3
  • 0.3.1
  • 0.3.0
  • 0.2.1
  • 0.2.0

Sphinx-Needs Ecosystem

In the last years, we have created additional information and extensions, which are based on or related to Sphinx-Needs:

Sphinx-Needs.com

The website presents the essential Sphinx-Needs functions and related extensions.

Also, it is a good entry point to understand the benefits and get an idea about the complete ecosystem of Sphinx-Needs.

Sphinx-Needs.com

Sphinx-Needs

Create, update, link, filter and present need objects like Requirements, Specifications, Bugs and many more.

The base extension provides all of its functionality under the MIT license for free.

Technical Docs

Sphinx-Needs Enterprise

Synchronize Sphinx-Needs data with external, company internal systems like CodeBeamer, Jira or Azure Boards.

Provides scripts to baseline data and makes CI usage easier.

Technical Docs

Sphinx-Test-Reports

Extension to import test results from XML files as need objects.

Created need objects can be filtered and linked to specification objects.

Technical Docs

Other Sphinx extensions

During the use of Sphinx-Needs in popular companies’ internal projects, we have created other Sphinx extensions to support the work of teams in the automotive industry:

Sphinx Collections

Extension to collect or generate files from different sources and include them in the Sphinx source folder.

It supports sources like Git repositories, Jinja based files or symlinks.

Technical Docs

Sphinx Bazel

Provides a Bazel domain in Sphinx documentation and allows the automated import of Bazel files and their documentation.

Technical Docs

Motivation

Sphinx-Test-Reports was created for an automotive project, which needs to document test results and their used environment configuration in an human-readable format. The goal is to provide enough information to be able to setup an identical test environment in 20+ years.

Sphinx-Test-Reports is part of a software bundle, which was designed to fulfill the parameters of the ISO 26262 standard for safety critical software in automotive companies.

Other tools are: sphinx-needs, sphinx-collections and tox-envreport.