On python3, Popen() universal_newlines=True converts the subprocess
stdout to unicode text using a codec based on user preferences. Given
LANG indicating ascii and utf-8 stdout from the subprocess, you'd get:
WARNING: kernel-doc '../scripts/kernel-doc -rst -enable-lineno
../drivers/media/dvb-core/demux.h' processing failed with: 'ascii' codec can't
decode byte 0xe2 in position 6368: ordinal not in range(128)
Fix this by dropping universal_newlines=True and replacing the implicit
LANG specific decode with an explicit utf-8 decode. This also gets rid
of the annoying conditional code for python 2 vs. 3.
Fixes: ba35018593 ("Documentation/sphinx: fix kernel-doc extension on python3")
Reference: http://mid.mail-archive.com/54c23e8e-89c0-5cea-0dcc-e938952c5642@infradead.org
Reported-and-tested-by: Randy Dunlap <rdunlap@infradead.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Instead of using 3 commands to install a virtualenv, use
a single one, reading the requirements from this file:
Documentation/sphinx/requirements.txt
Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
There were a few bits and pieces left over from the now-disused DocBook
toolchain; git rid of them.
Reported-by: Markus Heiser <markus.heiser@darmarit.de>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Currently, the script just assumes to be called at
Documentation/sphinx/. Change it to work on any directory,
and make it abort if something gets wrong.
Also, be sure that both parameters are specified.
That should avoid troubles like this:
$ Documentation/sphinx/tmplcvt Documentation/DocBook/writing_usb_driver.tmpl
sed: couldn't open file convert_template.sed: No such file or directory
Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Using the development version of sphinx caused the parsing of the
version to fail.
Signed-off-by: Rémy Léone <remy.leone@gmail.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
In docutils 0.13, the return type of get_column_widths method of the
Table directive has changed [1], which breaks our flat-table directive
and leads to a TypeError when trying to build the docs [2].
This patch adds support for the new return type, while keeping support
for older docutils versions too.
[1] https://sourceforge.net/p/docutils/patches/120/
[2] https://sourceforge.net/p/docutils/bugs/303/
Signed-off-by: Dmitry Shachnev <mitya57@debian.org>
Cc: <stable@vger.kernel.org> # 4.8.x-
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Keeping both rst and in-file documentation in sync can be harsh.
So, simplify the script's internal documntation to a bare minimum,
and add a mention to the ReST file with its full documentation.
This way, a quick help is still available at the command line,
while the complete one is maintained at the ReST format.
As we won't be using pad2rst anymore, do a cleanup at the ReST
file.
Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Provide a man page for parse-headers.pl, describing
how to use it.
The documentation on ReST format was generated via pod2rst:
http://search.cpan.org/~dowens/Pod-POM-View-Restructured-0.02/bin/pod2rst
Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Python module names should not have hyphens per [PEP 8]. Drop the hyphen
from kernel-doc.py. The extension directive remains unchanged.
[PEP 8] https://www.python.org/dev/peps/pep-0008/#package-and-module-names
Reported-by: Markus Heiser <markus.heiser@darmarit.de>
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Merge back from docs-next in order to get the cdomain extension.
With such extension, the number of warnings when building docs
in nitpick mode reduced from 22 to 2 warnings.
* docs-next/docs-next:
docs/driver-model: fix typo
DMA-API-HOWTO: <asm/generic/scatterlist.h> is no more
doc-rst:c-domain: function-like macros arguments
doc-rst:c-domain: fix sphinx version incompatibility
Documentation/filesystems: Fixed typo
docs: Don't format internal MPT docs
docs: split up serial-interfaces.rst
docs: Pull the HSI documentation together
docs: Special-case function-pointer parameters in kernel-doc
docs: make kernel-doc handle varargs properly
x86: fix memory ranges in mm documentation
documentation/scsi: Remove nodisconnect parameter
doc: ioctl: Add some clarifications to botching-up-ioctls
docs: split up the driver book
Docs: sphinxify device-drivers.tmpl
Handle signatures of function-like macros well. Don't try to deduce
arguments types of function-like macros.
Signed-off-by: Markus Heiser <markus.heiser@darmarIT.de>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
The self.indexnode's tuple has changed in sphinx version 1.4, from a
former 4 element tuple to a 5 element tuple.
e6a5a3a92e
Signed-off-by: Markus Heiser <markus.heiser@darmarIT.de>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Instead of keep using the normal reference, move to the C
domain ones. Using C domains everywhere will allow
cross-references between kAPI and uAPI docs.
Acked-by: Jonathan Corbet <corbet@lwn.net>
Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Add a parser for the --debug option, in order to allow
seeing what the parser is doing.
Acked-by: Jonathan Corbet <corbet@lwn.net>
Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
The setup() function of a Sphinx-extension can return a dictionary. This
is treated by Sphinx as metadata of the extension [1].
With metadata "parallel_read_safe = True" a extension is marked as
save for "parallel reading of source". This is needed if you want
build in parallel with N processes. E.g.:
make SPHINXOPTS=-j4 htmldocs
will no longer log warnings like:
WARNING: the foobar extension does not declare if it is safe for
parallel reading, assuming it isn't - please ask the extension author
to check and make it explicit.
Add metadata to extensions:
* kernel-doc
* flat-table
* kernel-include
[1] http://www.sphinx-doc.org/en/stable/extdev/#extension-metadata
Signed-off-by: Markus Heiser <markus.heiser@darmarIT.de>
Tested-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Moved the *duplicate C object description* warnings for function
declarations in the nitpicky mode. In nitpick mode, you can suppress
those warnings (e.g. ioctl) with::
nitpicky = True
nitpick_ignore = [
("c:func", "ioctl"),
]
See Sphinx documentation for the config values for ``nitpick`` and
``nitpick_ignore`` [1].
With this change all the ".. cpp:function:: int ioctl(..)" descriptions
(found in the media book) can be migrated to ".. c:function:: int
ioctl(..)", without getting any warnings. E.g.::
.. cpp:function:: int ioctl( int fd, int request, struct cec_event *argp )
.. c:function:: int ioctl( int fd, int request, struct cec_event *argp )
The main effect, is that we get those *CPP-types* back into Sphinx's C-
namespace and we need no longer to distinguish between c/cpp references,
when we refer a function like the ioctl.
[1] http://www.sphinx-doc.org/en/stable/config.html?highlight=nitpick#confval-nitpicky
Signed-off-by: Markus Heiser <markus.heiser@darmarIT.de>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Add option 'name' to the "c:function:" directive. With option 'name'
the ref-name of a function can be modified. E.g.::
.. c:function:: int ioctl( int fd, int request )
:name: VIDIOC_LOG_STATUS
The func-name (e.g. ioctl) remains in the output but the ref-name
changed from ``ioctl`` to ``VIDIOC_LOG_STATUS``. The index entry for
this function is also changed to ``VIDIOC_LOG_STATUS`` and the function
can now referenced by::
:c:func:`VIDIOC_LOG_STATUS`
Signed-off-by: Markus Heiser <markus.heiser@darmarIT.de>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Add a sphinx-extension to customize the sphinx c-domain. No functional
changes right yet, just the boilerplate code.
Signed-off-by: Markus Heiser <markus.heiser@darmarIT.de>
[ jc: coding-style tweak ]
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Let's escape the LaTeX characters, to avoid troubles when
outputing them.
Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Add a generic way to build only a reST sub-folder with or
without a individual *build-theme*.
* control *sub-folders* by environment SPHINXDIRS
* control *build-theme* by environment SPHINX_CONF
Folders with a conf.py file, matching $(srctree)/Documentation/*/conf.py
can be build and distributed *stand-alone*. E.g. to compile only the
html of 'media' and 'gpu' folder use::
make SPHINXDIRS="media gpu" htmldocs
To use an additional sphinx-build configuration (*build-theme*) set the
name of the configuration file to SPHINX_CONF. E.g. to compile only the
html of 'media' with the *nit-picking* build use::
make SPHINXDIRS=media SPHINX_CONF=conf_nitpick.py htmldocs
With this, the Documentation/conf.py is read first and updated with the
configuration values from the Documentation/media/conf_nitpick.py.
Signed-off-by: Markus Heiser <markus.heiser@darmarIT.de>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
* 'docs-next' of git://git.lwn.net/linux:
doc-rst: kernel-doc: fix handling of address_space tags
Revert "doc/sphinx: Enable keep_warnings"
doc-rst: kernel-doc directive, fix state machine reporter
docs: deprecate kernel-doc-nano-HOWTO.txt
doc/sphinx: Enable keep_warnings
Documentation: add watermark_scale_factor to the list of vm systcl file
kernel-doc: Fix up warning output
docs: Get rid of some kernel-documentation warnings
Add a reporter replacement that assigns the correct source name and line
number to a system message, as recorded in a ViewList.
[1] http://mid.gmane.org/CAKMK7uFMQ2wOp99t-8v06Om78mi9OvRZWuQsFJD55QA20BB3iw@mail.gmail.com
Signed-off-by: Markus Heiser <markus.heiser@darmarIT.de>
Tested-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
The function that replace references add a "\ " at the end of
references, to avoid the ReST markup parser to not identify
them as references. That works fine except for the end of lines,
as a sequence of { '\', ' ', '\n' } characters makes Sphinx
to ignore the end of line. So, strip those escape/spaces at the
end of lines.
Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
The kernel-include directive is needed to include the auto generated rst
content from a build (pre-) process. E.g. the linux_tv Makefile
generates intermediate reST-files from header files. Since there is a O=
option:
make O=dir [targets] Locate all output files in "dir"
We need to include intermediate reST files from arbitrary (O=/tmp/foo)
locations:
The 'kernel-include' reST-directive is a replacement for the 'include'
directive. The 'kernel-include' directive expand environment variables
in the path name and allows to include files from arbitrary locations.
.. hint::
Including files from arbitrary locations (e.g. from '/etc') is a
security risk for builders. This is why the 'include' directive from
docutils *prohibit* pathnames pointing to locations *above* the
filesystem tree where the reST document with the include directive is
placed.
Substrings of the form $name or ${name} are replaced by the value of
environment variable name. Malformed variable names and references to
non-existing variables are left unchanged.
Signed-off-by: Markus Heiser <markus.heiser@darmarIT.de>
Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Add one extra escape character to avoid those warnings:
Documentation/linux_tv/videodev2.h.rst:6: WARNING: Inline substitution_reference start-string without end-string.
Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
At videodev2.h, we have hundreds of symbols that don't
currently have a reference yet. Let's ignore for how, while
we don't improve those cross-refs.
Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
We should not let comments to mangle with the symbols
parsing. Unfortunately, videodev2.h has lots of those
in the middle of enums and structs. So, we need to improve
our parser to discard them.
Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
The typedef handler should do two things to be generic:
1) parse typedef enums;
2) accept both possible syntaxes:
typedef struct foo { .. } foo_t;
typedef struct { .. } foo_t;
Unfortunately, this is needed to parse some legacy DVB
files, like dvb/audio.h.
Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
When typedef is used on its multiline format, we need to
also parse enum and struct in the same line.
Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Be more formal about the valid symbols that are expected by
the parser, to match what c language expects.
Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
The dmx.h header has two things that causes the parser to
break while handling enums:
per-header enums and the '{' starts on a new line
Both makes the parser to get lexical marks to be detected
as if they were symbols.
Fix it.
Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
As we had to escape the symbols for the ReST markup to not do
the wrong thing, the logic to discover start/end of strings
are not trivial. Improve the end delimiter detection, in order
to highlight more occurrences of the strings.
Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
This script parses a header file and converts it into a
parsed-literal block, creating references for ioctls,
defines, typedefs, enums and structs.
It also allow an external file to modify the rules, in
order to fix the expressions.
Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Implements the reST flat-table directive.
The ``flat-table`` is a double-stage list similar to the ``list-table`` with
some additional features:
* column-span: with the role ``cspan`` a cell can be extended through
additional columns
* row-span: with the role ``rspan`` a cell can be extended through
additional rows
* auto span rightmost cell of a table row over the missing cells on the right
side of that table-row. With Option ``:fill-cells:`` this behavior can
changed from *auto span* to *auto fill*, which automaticly inserts (empty)
list tables
The *list tables* formats are double stage lists. Compared to the
ASCII-art they migth be less comfortable for readers of the
text-files. Their advantage is, that they are easy to create/modify
and that the diff of a modification is much more meaningfull, because
it is limited to the modified content.
The initial implementation was taken from the sphkerneldoc project [1]
[1] https://github.com/return42/sphkerneldoc/commits/master/scripts/site-python/linuxdoc/rstFlatTable.py
Signed-off-by: Markus Heiser <markus.heiser@darmarIT.de>
[jc: fixed typos and misspellings in the docs]
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Let the user specify file patterns where to look for the EXPORT_SYMBOLs
in addition to the file with kernel-doc comments. This is directly based
on the -export-file FILE option added to kernel-doc in "kernel-doc: add
support for specifying extra files for EXPORT_SYMBOLs", but we extend
that with globbing patterns in the Sphinx extension.
The file patterns are added as options to the :export: and :internal:
arguments of the kernel-doc directive. For example, to extract the
documentation of exported functions from include/net/mac80211.h:
.. kernel-doc:: include/net/mac80211.h
:export: net/mac80211/*.c
Without the file pattern, no exported functions would be found, as the
EXPORT_SYMBOLs are placed in the various source files under
net/mac80211.
The matched files are also added as dependencies on the document in
Sphinx, as they may affect the output. This is one of the reasons to do
the globbing in the Sphinx extension instead of in scripts/kernel-doc.
The file pattern remains optional, and is not needed if the kernel-doc
comments and EXPORT_SYMBOLs are placed in the source file passed in as
the main argument to the kernel-doc directive. This is the most common
case across the kernel source tree.
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
Design is pretty simple: kernel-doc inserts breadcrumbs with line
numbers, and sphinx picks them up. At first I went with a sphinx
comment, but inserting those at random places seriously upsets the
parser, and must be filtered. Hence why this version now uses "#define
LINEO " since one of these ever escape into output it's pretty clear
there is a bug.
It seems to work well, and at least the 2-3 errors where sphinx
complained about something that was not correct in kernel-doc text the
line numbers matched up perfectly.
v2: Instead of noodling around in the parser state machine, create
a ViewList and parse it ourselves. This seems to be the recommended
way, per Jani's suggestion.
v3:
- Split out ViewList pach. Splitting the kernel-doc changes from the
sphinx ones isn't possible, since emitting the LINENO lines wreaks
havoc with the rst formatting. We must filter them.
- Improve the regex per Jani's suggestions, and compile it just once
for speed.
- Now that LINENO lines are eaten, also add them to function parameter
descriptions. Much less content and offset than for in-line struct
member descriptions, but still nice to know which exact continuation
line upsets sphinx.
- Simplify/clarify the line +/-1 business a bit.
v4: Split out the scripts/kernel-doc changes and make line-numbers
opt-in, as suggested by Jani.
Cc: Jani Nikula <jani.nikula@intel.com>
Cc: linux-doc@vger.kernel.org
Cc: Jonathan Corbet <corbet@lwn.net>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
Instead of just forcefully inserting our kernel-doc input and letting
the state machine stumble over it the recommended way is to create
ViewList, parse that and then return the list of parsed nodes.
Suggested by Jani.
Cc: Jani Nikula <jani.nikula@intel.com>
Cc: linux-doc@vger.kernel.org
Cc: Jonathan Corbet <corbet@lwn.net>
Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
With this error output becomes almost readable. The line numbers are
still totally bonghits, but that's a lot harder to pull out of
kerneldoc. We'd essentially have to insert some special markers in the
kernel-doc output, split the output along these markers and then
insert each block separately using
state_machine.insert_input(block, source, first_line)
Cc: Jani Nikula <jani.nikula@intel.com>
Cc: linux-doc@vger.kernel.org
Cc: Jonathan Corbet <corbet@lwn.net>
Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
Reconcile differences between python2 and python3 on dealing with
stdout, stderr from Popen. This fixes "name 'unicode' is not defined"
errors on python3. We'll need to try to keep the extension working on
both python-sphinx and python3-sphinx so we don't need two copies.
Reported-and-tested-by: Marius Vlad <marius.c.vlad@intel.com>
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
This script uses pandoc to convert existing DocBook template files to RST
templates. A couple of sed scripts are need to massage things both before
and after the conversion, but the result is then usable with no hand
editing.
[Jani: Change usage to tmplcvt <in> <out>. Fix escaping for docproc
directives. Add support the new kernel-doc extension.]
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
Add an extension to handle kernel-doc directives, to call kernel-doc
according to the arguments and parameters given to the reStructuredText
directive.
The syntax for the kernel-doc directive is:
.. kernel-doc:: FILENAME
:export:
:internal:
:functions: FUNCTION [FUNCTION ...]
:doc: SECTION TITLE
Of the directive options export, internal, functions, and doc, currently
only one option may be given at a time.
The FILENAME is relative from the kernel source tree root.
The extension notifies Sphinx about the document dependency on FILENAME,
causing the document to be rebuilt when the file has been changed.
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
