test-report

test-report generate a complete report including test-cases, tables and statistics for a given test-file.

.. test-report:: My Report
   :id: REPORT
   :file: ../tests/doc_test/utils/pytest_sphinx_data_short.xml

The following options must be set:

  • id: An unique id. Will be used as prefix for all created objects.

  • file: File path to the test file.

Optional options are:

  • tags: comma separated list of tags. Will be set to all created test objects.

  • links: comma separated list of links. Will be set to all created test objects.

Example

Used code:

.. test-report:: My Report
   :id: REPORT
   :file: ../tests/doc_test/utils/pytest_sphinx_data_short.xml
   :tags: my_report, awesome
   :links: SPEC_001

.. spec:: Example specification
   :id: SPEC_001

   Used as simple link target. See Sphinx-Needs for details.

Result:

Test file: /home/docs/checkouts/readthedocs.org/user_builds/sphinx-test-reports/checkouts/latest/docs/../tests/doc_test/utils/pytest_sphinx_data_short.xml

Template used: /home/docs/checkouts/readthedocs.org/user_builds/sphinx-test-reports/envs/latest/lib/python3.7/site-packages/sphinxcontrib/test_reports/directives/test_report_template.txt

Statistics

Test suites: 1
Test cases: 7
Failed test cases: 2
Skipped test cases: 1

Test cases:

ID

Title

Result

REPORT_0EA_049F6

test_insert_similar

passed

REPORT_0EA_140FB

test_rstdim_to_latexdim

passed

REPORT_0EA_57BA7

test_theme_api

failure

REPORT_0EA_734D6

test_build

passed

REPORT_0EA_73FAE

test_build_latex_doc[lualatex-howto]

failure

REPORT_0EA_8F625

test_generate

passed

REPORT_0EA_EB670

test_texinfo

skipped

Failed test cases:

ID

Title

Result

REPORT_0EA_57BA7

test_theme_api

failure

REPORT_0EA_73FAE

test_build_latex_doc[lualatex-howto]

failure

Skipped test cases:

ID

Title

Result

REPORT_0EA_EB670

test_texinfo

skipped

Imported data

Test-File: My Report REPORT ../_images/arrow-right-circle.svg
tags: my_report, awesome, REPORT
delete: False
jinja_content: False
file: /home/docs/checkouts/readthedocs.org/user_builds/sphinx-test-reports/checkouts/latest/docs/../tests/doc_test/utils/pytest_sphinx_data_short.xml
suites: 1
cases: 7
passed: 4
skipped: 1
failed: 2
errors: 0

[‘This file contains a subset of executed tests for Sphinx.’]

Test-Suite: pytest REPORT_0EA ../_images/arrow-right-circle.svg
tags: my_report, awesome, REPORT
delete: False
jinja_content: False
file: /home/docs/checkouts/readthedocs.org/user_builds/sphinx-test-reports/checkouts/latest/docs/../tests/doc_test/utils/pytest_sphinx_data_short.xml
suite: pytest
cases: 7
passed: 4
skipped: 1
failed: 2
errors: 0
Test-Case: test_insert_similar REPORT_0EA_049F6 ../_images/arrow-right-circle.svg
tags: my_report, awesome, REPORT
style: tr_passed
delete: False
jinja_content: False
file: /home/docs/checkouts/readthedocs.org/user_builds/sphinx-test-reports/checkouts/latest/docs/../tests/doc_test/utils/pytest_sphinx_data_short.xml
suite: pytest
case: test_insert_similar
case_name: test_insert_similar
classname: tests.test_versioning
time: 0.02409982681274414
result: passed
links outgoing: SPEC_001, REPORT, REPORT_0EA
Test-Case: test_build REPORT_0EA_734D6 ../_images/arrow-right-circle.svg
tags: my_report, awesome, REPORT
style: tr_passed
delete: False
jinja_content: False
file: /home/docs/checkouts/readthedocs.org/user_builds/sphinx-test-reports/checkouts/latest/docs/../tests/doc_test/utils/pytest_sphinx_data_short.xml
suite: pytest
case: test_build
case_name: test_build
classname: tests.test_websupport
time: 0.3648707866668701
result: passed
links outgoing: SPEC_001, REPORT, REPORT_0EA

System-out:

#x1B[01mRunning Sphinx v1.7+/aecb0082#x1B[39;49;00m
#x1B[01mloading pickled environment... #x1B[39;49;00mnot yet created
#x1B[01mbuilding [mo]: #x1B[39;49;00mtargets for 0 po files that are out of date
#x1B[01mbuilding [websupport]#x1B[39;49;00m: targets for 1 source files that are out of date
#x1B[01mupdating environment: #x1B[39;49;00m1 added, 0 changed, 0 removed
Test-Case: test_rstdim_to_latexdim REPORT_0EA_140FB ../_images/arrow-right-circle.svg
tags: my_report, awesome, REPORT
style: tr_passed
delete: False
jinja_content: False
file: /home/docs/checkouts/readthedocs.org/user_builds/sphinx-test-reports/checkouts/latest/docs/../tests/doc_test/utils/pytest_sphinx_data_short.xml
suite: pytest
case: test_rstdim_to_latexdim
case_name: test_rstdim_to_latexdim
classname: tests.test_writer_latex
time: 0.0007483959197998047
result: passed
links outgoing: SPEC_001, REPORT, REPORT_0EA
Test-Case: test_generate REPORT_0EA_8F625 ../_images/arrow-right-circle.svg
tags: my_report, awesome, REPORT
style: tr_passed
delete: False
jinja_content: False
file: /home/docs/checkouts/readthedocs.org/user_builds/sphinx-test-reports/checkouts/latest/docs/../tests/doc_test/utils/pytest_sphinx_data_short.xml
suite: pytest
case: test_generate
case_name: test_generate
classname: tests.py35.test_autodoc_py35
time: 0.3316631317138672
result: passed
links outgoing: SPEC_001, REPORT, REPORT_0EA
Test-Case: test_theme_api REPORT_0EA_57BA7 ../_images/arrow-right-circle.svg
tags: my_report, awesome, REPORT
style: tr_failure
delete: False
jinja_content: False
file: /home/docs/checkouts/readthedocs.org/user_builds/sphinx-test-reports/checkouts/latest/docs/../tests/doc_test/utils/pytest_sphinx_data_short.xml
suite: pytest
case: test_theme_api
case_name: test_theme_api
classname: tests.test_theming
time: 0.01364278793334961
result: failure
links outgoing: SPEC_001, REPORT, REPORT_0EA

Text:

app = <SphinxTestApp buildername='html'>, status = <_io.StringIO
object at 0x7ff4b6c574c8>, warning = <_io.StringIO object at 0x7ff4b6c57798>

@pytest.mark.sphinx(
confoverrides={'html_theme': 'ziptheme',
'html_theme_options.testopt': 'foo'})
def test_theme_api(app, status, warning):
E Extra items in the left set:
E 'alabaster'
E Use -v to get the full diff

tests/test_theming.py:26: AssertionError

System-out:

# testroot: root
# builder: html
# srcdir: /tmp/root
# outdir: /tmp/root/_build/html
# status:
# warning:
Test-Case: test_build_latex_doc[lualatex-howto] REPORT_0EA_73FAE ../_images/arrow-right-circle.svg
tags: my_report, awesome, REPORT
style: tr_failure
delete: False
jinja_content: False
file: /home/docs/checkouts/readthedocs.org/user_builds/sphinx-test-reports/checkouts/latest/docs/../tests/doc_test/utils/pytest_sphinx_data_short.xml
suite: pytest
case: test_build_latex_doc[lualatex-howto]
case_name: test_build_latex_doc
case_parameter: lualatex-howto
classname: tests.test_build_latex
time: 0.9610209465026855
result: failure
links outgoing: SPEC_001, REPORT, REPORT_0EA

Text:

app = <SphinxTestApp buildername='latex'>, status = <_io.StringIO object at
0x7ff4c662a438>, warning = <_io.StringIO object at 0x7ff4c662a678>
engine = 'lualatex', docclass = 'howto'

@skip_if_requested
@skip_if_stylefiles_notfound
print(stdout)
print(stderr)
> assert False, '%s exited with return code %s' % (
app.config.latex_engine, p.returncode)
E AssertionError: lualatex exited with return code 1
E assert False

tests/test_build_latex.py:86: AssertionError

System-out:

b"This is LuaTeX, Version 1.0.4 (TeX Live 2017/Debian) \n restricted system commands
enabled.\n(./lualatex/SphinxTests.tex\nLaTeX2e <2017-04-15>\n[\\directlua]:1: module 'luaotfload-main'
not found:\n\tno field package.preload['luaotfload-main']\n\t[kpse lua searcher] file not found:
#x1B[31m/tmp/root/contents.txt:10: WARNING: extensions is already assigned section numbers (nested numbered
toctree?)#x1B[39;49;00m
#x1B[31m/tmp/root/robots.txt: WARNING: document isn't included in any toctree#x1B[39;49;00m
Test-Case: test_texinfo REPORT_0EA_EB670 ../_images/arrow-right-circle.svg
tags: my_report, awesome, REPORT
style: tr_skipped
delete: False
jinja_content: False
file: /home/docs/checkouts/readthedocs.org/user_builds/sphinx-test-reports/checkouts/latest/docs/../tests/doc_test/utils/pytest_sphinx_data_short.xml
suite: pytest
case: test_texinfo
case_name: test_texinfo
classname: tests.test_build_texinfo
time: 0.6067860126495361
result: skipped
links outgoing: SPEC_001, REPORT, REPORT_0EA

Text:

/home/daniel/workspace/sphinx/sphinx/tests/test_build_texinfo.py:61: <py._xmlgen.raw object at
0x7ff4c6fc0978>

Message:

<Skipped instance>

System-out:

# testroot: root
# builder: texinfo
# srcdir: /tmp/root
# outdir: /tmp/root/_build/texinfo
# status:
#x1B[01mRunning Sphinx v1.7+/aecb0082#x1B[39;49;00m
#x1B[01mloading pickled environment... #x1B[39;49;00mdone

Template

The following rst-code is used to generate the report.

**Test file**: {file}

**Template used**: {template_path}

**Statistics**

| Test suites: :need_count:`'{id}' in tags and type=='{suite_need}'`
| Test cases: :need_count:`'{id}' in tags and type=='{case_need}'`
| Failed test cases: :need_count:`'{id}' in tags and 'failure' == result and type=='{case_need}'`
| Skipped test cases: :need_count:`'{id}' in tags and 'skipped' == result and type=='{case_need}'`

**Test cases**:

.. needtable::
   :filter: '{id}' in tags and type == '{case_need}'
   :columns: id, title, result
   :style_row: tr_[[copy('result')]]

**Failed test cases**:

.. needtable::
   :filter: '{id}' in tags and 'failure' == result
   :columns: id, title, result
   :style_row: tr_[[copy('result')]]

**Skipped test cases**:

.. needtable::
   :filter: '{id}' in tags and 'skipped' == result
   :columns: id, title, result
   :style_row: tr_[[copy('result')]]

**Imported data**


.. {file_type}:: {title}
   :id: {id}{links_string}
   :tags: {tags}
   :file: {file}
   :auto_suites:
   :auto_cases:

   {content}


Template
--------
The following rst-code is used to generate the report.

.. literalinclude:: ../../sphinxcontrib/test_reports/directives/test_report_template.txt
   :language: rst
Specification: Example specification SPEC_001 ../_images/arrow-right-circle.svg

Used as simple link target. See Sphinx-Needs for details.