test-file

test_file loads the data from a given file path in file.

Usage

.. test-file:: My Test Data
   :file: my_test_data.xml
   :id: TESTFILE_1

The following options can be set:

  • id: Unique id for the test file. If not given, generated from title.

  • file: file path to test file. If relative, the location of conf.py folder is taken as base-folder.

  • status: A status as string.

  • tags: A comma-separated list of strings.

  • links: A comma-separated list of IDs to other documented test_files / needs-objects.

  • collapse: If set to “TRUE”, meta data is collapsed. Can also be set to “FALSE”.

  • auto_suites: If set (no value!), all contained suites will get documented (like using test-suite).

  • auto_cases: If set (no value!), all contained cases will get documented (like using test-cases). Must be used together with auto_suites!

test_file creates a need of type Test-File and adds the following options automatically:

  • suites: Amount of found suites in test file.

  • cases: Amount of found cases in test file.

  • passed: Amount of passed test cases.

  • skipped: Amount of skipped test cases.

  • failed: Amount of failed test cases.

  • errors: Amount of test cases which have errors during tet execution.

These options can also be used to filter for certain Test-Files.

Autogenerated suites and cases

By using the options auto_suites or auto_cases Sphinx-Test-Report will documented all found suites/cases in the given test-file.

So it will create sphinx-needs-objects, like you would manually use the directives test-suite or test-case.

It uses the ID of test_file as prefix for the IDs of the suites and cases. Their ID will be hash-value based on their title. So a suite ID may look like TESTFILE1_E2A5 and a case ID TESTFILE1_E2A5_F39A.

These IDs should be stable between different sphinx builds, as long as their title of suites / cases does not change.

All other options like tags, links, status are copied from test-file directive to the autogenerated test-suite and test-case directives.

links will also contain links to the parent objects, e.g. a test-case has links to the test_suite and also to the test-file.

Example

Basics

.. test-file:: common xml test data
   :file: ../tests/doc_test/utils/xml_data.xml
   :id: TESTFILE_1

   This test_file has very common data.
   Some options are net set, therefore their value is ``-1``

.. test-file:: pytest test data
   :file: ../tests/doc_test/utils/pytest_data.xml
   :id: TESTFILE_2
   :links: TESTFILE_1

   This test_file was created by `pytest <https://docs.pytest.org/en/latest/>`_.

.. test-file:: nose test data
   :file: ../tests/doc_test/utils/nose_data.xml
   :id: TESTFILE_3
   :links: TESTFILE_1
   :status: open
   :tags: pytest, data, awesome
   :collapse: FALSE

   This test_file was created by `nosetest <https://nose.readthedocs.io/en/latest/>`_.

   ``collapse`` was set to False, therefor we see its data directly.
   Also ``status`` and ``tags`` are set.
Test-File: common xml test data TESTFILE_1 ../_images/arrow-right-circle.svg
doctype: .rst
delete: False
jinja_content: False
file: ../tests/doc_test/utils/xml_data.xml
suites: 1
cases: 3
passed: 3
skipped: -1
failed: -1
errors: -1
links incoming: TESTFILE_2, TESTFILE_3

This test_file has very common data. Some options are not set, therefore their value is -1
Test-File: pytest test data TESTFILE_2 ../_images/arrow-right-circle.svg
doctype: .rst
delete: False
jinja_content: False
file: ../tests/doc_test/utils/pytest_data.xml
suites: 1
cases: 10
passed: 0
skipped: 10
failed: 0
errors: 0
links outgoing: TESTFILE_1
links incoming: TESTSUITE_1

This test_file was created by pytest.
Test-File: nose test data TESTFILE_3 ../_images/arrow-right-circle.svg
doctype: .rst
status: open
tags: pytest, data, awesome
delete: False
jinja_content: False
file: ../tests/doc_test/utils/nose_data.xml
suites: 1
cases: 5
passed: 5
skipped: 0
failed: 0
errors: 0
links outgoing: TESTFILE_1

This test_file was created by nosetest.

collapse was set to False, therefor we see its data directly. Also status and tags are set.

Autogenerated suite

.. test-file:: pytest test data
   :file: ../tests/doc_test/utils/pytest_data.xml
   :id: TESTFILE_AUTO1
   :auto_suites:
   :collapse: FALSE
   :tags: auto
Test-File: pytest test data TESTFILE_AUTO1 ../_images/arrow-right-circle.svg
doctype: .rst
tags: auto
delete: False
jinja_content: False
file: ../tests/doc_test/utils/pytest_data.xml
suites: 1
cases: 10
passed: 0
skipped: 10
failed: 0
errors: 0
links incoming: TESTFILE_AUTO1_0EA
Test-Suite: pytest TESTFILE_AUTO1_0EA ../_images/arrow-right-circle.svg
tags: auto
delete: False
jinja_content: False
file: ../tests/doc_test/utils/pytest_data.xml
suite: pytest
cases: 10
passed: 0
skipped: 10
failed: 0
errors: 0
links outgoing: TESTFILE_AUTO1

Autogenerated suite and cases

.. test-file:: pytest test data
   :file: ../tests/doc_test/utils/pytest_data.xml
   :id: TESTFILE_AUTO2
   :auto_suites:
   :auto_cases:
   :tags: auto
Test-File: pytest test data TESTFILE_AUTO2 ../_images/arrow-right-circle.svg
doctype: .rst
tags: auto
delete: False
jinja_content: False
file: ../tests/doc_test/utils/pytest_data.xml
suites: 1
cases: 10
passed: 0
skipped: 10
failed: 0
errors: 0
links incoming: TESTFILE_AUTO2_0EA, TESTFILE_AUTO2_0EA_D962E, TESTFILE_AUTO2_0EA_BCA04, TESTFILE_AUTO2_0EA_5B0AB, TESTFILE_AUTO2_0EA_17650, TESTFILE_AUTO2_0EA_44574, TESTFILE_AUTO2_0EA_52AB6, TESTFILE_AUTO2_0EA_03B18, TESTFILE_AUTO2_0EA_D9167, TESTFILE_AUTO2_0EA_8E5DB, TESTFILE_AUTO2_0EA_E5144
Test-Suite: pytest TESTFILE_AUTO2_0EA ../_images/arrow-right-circle.svg
tags: auto
delete: False
jinja_content: False
file: ../tests/doc_test/utils/pytest_data.xml
suite: pytest
cases: 10
passed: 0
skipped: 10
failed: 0
errors: 0
links outgoing: TESTFILE_AUTO2
links incoming: TESTFILE_AUTO2_0EA_D962E, TESTFILE_AUTO2_0EA_BCA04, TESTFILE_AUTO2_0EA_5B0AB, TESTFILE_AUTO2_0EA_17650, TESTFILE_AUTO2_0EA_44574, TESTFILE_AUTO2_0EA_52AB6, TESTFILE_AUTO2_0EA_03B18, TESTFILE_AUTO2_0EA_D9167, TESTFILE_AUTO2_0EA_8E5DB, TESTFILE_AUTO2_0EA_E5144
Test-Case: FLAKE8 TESTFILE_AUTO2_0EA_D962E ../_images/arrow-right-circle.svg
tags: auto
style: tr_skipped
delete: False
jinja_content: False
file: ../tests/doc_test/utils/pytest_data.xml
suite: pytest
case: FLAKE8
case_name: FLAKE8
classname: setup
time: 0.000252246856689
result: skipped
links outgoing: TESTFILE_AUTO2, TESTFILE_AUTO2_0EA

Text:

/home/daniel/workspace/sphinx/sphinx-test-reports/.tox/py27-sphinx15/local/lib/python2.7/site-packages/pytest_flake8.py:106: <py._xmlgen.raw object at 0x7fd5a8a0e950>

Message:

file(s) previously passed FLAKE8 checks
Test-Case: FLAKE8 TESTFILE_AUTO2_0EA_BCA04 ../_images/arrow-right-circle.svg
tags: auto
style: tr_skipped
delete: False
jinja_content: False
file: ../tests/doc_test/utils/pytest_data.xml
suite: pytest
case: FLAKE8
case_name: FLAKE8
classname: docs.conf
time: 0.000167846679688
result: skipped
links outgoing: TESTFILE_AUTO2, TESTFILE_AUTO2_0EA

Text:

/home/daniel/workspace/sphinx/sphinx-test-reports/.tox/py27-sphinx15/local/lib/python2.7/site-packages/pytest_flake8.py:106: <py._xmlgen.raw object at 0x7fd5a8a0ea10>

Message:

file(s) previously passed FLAKE8 checks
Test-Case: FLAKE8 TESTFILE_AUTO2_0EA_5B0AB ../_images/arrow-right-circle.svg
tags: auto
style: tr_skipped
delete: False
jinja_content: False
file: ../tests/doc_test/utils/pytest_data.xml
suite: pytest
case: FLAKE8
case_name: FLAKE8
classname: sphinxcontrib.__init__
time: 0.000164031982422
result: skipped
links outgoing: TESTFILE_AUTO2, TESTFILE_AUTO2_0EA

Text:

/home/daniel/workspace/sphinx/sphinx-test-reports/.tox/py27-sphinx15/local/lib/python2.7/site-packages/pytest_flake8.py:106: <py._xmlgen.raw object at 0x7fd5a8497450>

Message:

file(s) previously passed FLAKE8 checks
Test-Case: FLAKE8 TESTFILE_AUTO2_0EA_17650 ../_images/arrow-right-circle.svg
tags: auto
style: tr_skipped
delete: False
jinja_content: False
file: ../tests/doc_test/utils/pytest_data.xml
suite: pytest
case: FLAKE8
case_name: FLAKE8
classname: sphinxcontrib.test_reports.__init__
time: 0.000192880630493
result: skipped
links outgoing: TESTFILE_AUTO2, TESTFILE_AUTO2_0EA

Text:

/home/daniel/workspace/sphinx/sphinx-test-reports/.tox/py27-sphinx15/local/lib/python2.7/site-packages/pytest_flake8.py:106: <py._xmlgen.raw object at 0x7fd5a8497c10>

Message:

file(s) previously passed FLAKE8 checks
Test-Case: FLAKE8 TESTFILE_AUTO2_0EA_44574 ../_images/arrow-right-circle.svg
tags: auto
style: tr_skipped
delete: False
jinja_content: False
file: ../tests/doc_test/utils/pytest_data.xml
suite: pytest
case: FLAKE8
case_name: FLAKE8
classname: sphinxcontrib.test_reports.junitparser
time: 0.00017786026001
result: skipped
links outgoing: TESTFILE_AUTO2, TESTFILE_AUTO2_0EA

Text:

/home/daniel/workspace/sphinx/sphinx-test-reports/.tox/py27-sphinx15/local/lib/python2.7/site-packages/pytest_flake8.py:106: <py._xmlgen.raw object at 0x7fd5a84a2350>

Message:

file(s) previously passed FLAKE8 checks
Test-Case: FLAKE8 TESTFILE_AUTO2_0EA_52AB6 ../_images/arrow-right-circle.svg
tags: auto
style: tr_skipped
delete: False
jinja_content: False
file: ../tests/doc_test/utils/pytest_data.xml
suite: pytest
case: FLAKE8
case_name: FLAKE8
classname: sphinxcontrib.test_reports.test_reports
time: 0.000218152999878
result: skipped
links outgoing: TESTFILE_AUTO2, TESTFILE_AUTO2_0EA

Text:

/home/daniel/workspace/sphinx/sphinx-test-reports/.tox/py27-sphinx15/local/lib/python2.7/site-packages/pytest_flake8.py:106: <py._xmlgen.raw object at 0x7fd5a84a2a50>

Message:

file(s) previously passed FLAKE8 checks
Test-Case: FLAKE8 TESTFILE_AUTO2_0EA_03B18 ../_images/arrow-right-circle.svg
tags: auto
style: tr_skipped
delete: False
jinja_content: False
file: ../tests/doc_test/utils/pytest_data.xml
suite: pytest
case: FLAKE8
case_name: FLAKE8
classname: sphinxcontrib.test_reports.directives.__init__
time: 0.000204801559448
result: skipped
links outgoing: TESTFILE_AUTO2, TESTFILE_AUTO2_0EA

Text:

/home/daniel/workspace/sphinx/sphinx-test-reports/.tox/py27-sphinx15/local/lib/python2.7/site-packages/pytest_flake8.py:106: <py._xmlgen.raw object at 0x7fd5a84a2cd0>

Message:

file(s) previously passed FLAKE8 checks
Test-Case: FLAKE8 TESTFILE_AUTO2_0EA_D9167 ../_images/arrow-right-circle.svg
tags: auto
style: tr_skipped
delete: False
jinja_content: False
file: ../tests/doc_test/utils/pytest_data.xml
suite: pytest
case: FLAKE8
case_name: FLAKE8
classname: sphinxcontrib.test_reports.directives.test_env
time: 0.000218868255615
result: skipped
links outgoing: TESTFILE_AUTO2, TESTFILE_AUTO2_0EA

Text:

/home/daniel/workspace/sphinx/sphinx-test-reports/.tox/py27-sphinx15/local/lib/python2.7/site-packages/pytest_flake8.py:106: <py._xmlgen.raw object at 0x7fd5a84ae250>

Message:

file(s) previously passed FLAKE8 checks
Test-Case: FLAKE8 TESTFILE_AUTO2_0EA_8E5DB ../_images/arrow-right-circle.svg
tags: auto
style: tr_skipped
delete: False
jinja_content: False
file: ../tests/doc_test/utils/pytest_data.xml
suite: pytest
case: FLAKE8
case_name: FLAKE8
classname: sphinxcontrib.test_reports.directives.test_report
time: 0.000199317932129
result: skipped
links outgoing: TESTFILE_AUTO2, TESTFILE_AUTO2_0EA

Text:

/home/daniel/workspace/sphinx/sphinx-test-reports/.tox/py27-sphinx15/local/lib/python2.7/site-packages/pytest_flake8.py:106: <py._xmlgen.raw object at 0x7fd5a84ae950>

Message:

file(s) previously passed FLAKE8 checks
Test-Case: FLAKE8 TESTFILE_AUTO2_0EA_E5144 ../_images/arrow-right-circle.svg
tags: auto
style: tr_skipped
delete: False
jinja_content: False
file: ../tests/doc_test/utils/pytest_data.xml
suite: pytest
case: FLAKE8
case_name: FLAKE8
classname: sphinxcontrib.test_reports.directives.test_results
time: 0.00019097328186
result: skipped
links outgoing: TESTFILE_AUTO2, TESTFILE_AUTO2_0EA

Text:

/home/daniel/workspace/sphinx/sphinx-test-reports/.tox/py27-sphinx15/local/lib/python2.7/site-packages/pytest_flake8.py:106: <py._xmlgen.raw object at 0x7fd5a84bf090>

Message:

file(s) previously passed FLAKE8 checks