Commit Graph

301 Commits

Author SHA1 Message Date
Simon Glass
ff75d6e03e buildman: Drop a TODO that is done
Buildman now uses worktrees when available, instead of doing a full clone.
This was done in this commit:

   76de29fc4f buildman: Use git worktrees instead of git clones when possible

Drop the TODO.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-08-05 11:47:56 -04:00
Simon Glass
7f275bdbb0 buildman: Drop a Python 2.7 comment
This is well out of date, but it is still reasonable to use a list. Drop
the comment.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-08-05 11:47:56 -04:00
Simon Glass
f2e6775cdd buildman: Allow lines without a symbol
The 'nm' tool can produce lines without a symbol, for example:

   00000004 t

Silently skip these and anything else without three fields. Drop the
warning since there is nothing the user can do about it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reported-by: Tom Rini <trini@konsulko.com>
2022-08-05 11:47:56 -04:00
Simon Glass
add76e7c44 buildman: Return an error if there are maintainer warnings
Detect warnings about missing maintain info and return result code 2 in
that case.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-08-05 11:47:56 -04:00
Simon Glass
256126c294 buildman: Replace the Options column with config name
This appears in boards.cfg but we want to remove it. Drop support for
generating it and reading it. Detect an old boards.cfg file that has
this field and regenerate it, to avoid problems.

Instead, add the config name in that place. This fixes a subtle bug in
the generation code, since it uses 'target' for the config name and then
overwrites the value in scan() by setting params['target'] to the name
of the defconfig. The defconfig name is not the same as the
SYS_CONFIG_NAME variable.

With this change, we still have the config name and it can be searched
by buildman, e.g. with:

   buildman -nv sun5i

Signed-off-by: Simon Glass <sjg@chromium.org>
Reported-by: Tom Rini <trini@konsulko.com>
2022-08-05 11:47:56 -04:00
Simon Glass
969fd333ba buildman: Tidy up pylint problems in boards module
Fix all the pylint warnings. Also tidy up the comments so that they show
type information, as required.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-08-05 11:47:56 -04:00
Simon Glass
a8a0141bc2 buildman: Incorporate the genboardscfg.py tool
Bring this tool into buildman, so we don't have to run it separately. The
board.cfg file is still produced as part of the build, to save time when
doing another build in the same working directory. If it is out of date
with respect to the Kconfig, it is updated.

Time to regenerate on a recent single-thread machine is 4.6s (1.3s on a
32-thread machine), so we do need some sort of cache if we want buildman
to be useful on incremental builds. We could use Python's pickle format
but:

- it seems useful to allow boards.cfg to be regenerated, at least for a
  while, in case other tools use it
- it is possible to grep the file easily, e.g. to find boards which use
  a particular SoC (similar to 'buildman -nv <soc>'

Signed-off-by: Simon Glass <sjg@chromium.org>
Suggested-by: Tom Rini <trini@konsulko.com>
2022-08-05 11:47:56 -04:00
Simon Glass
c52bd22539 buildman: Split out Boards into its own file
Use a separate file for the Boards class so that its name matches the
module name.

Fix up the function names to match the pylint style and fix some other
warnings.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-08-05 11:47:56 -04:00
Simon Glass
6014db68d3 buildman: Convert camel case in board.py
Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-08-05 11:47:56 -04:00
Simon Glass
fb5cb07abe buildman: Drop use of 'board' in board module
Use brds instead so that we can reserve 'boards' and 'board' as module
names.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-08-05 11:47:55 -04:00
Simon Glass
cc2c0d1800 buildman: Drop use of 'boards' in control
Use brds instead so that we can reserve 'boards' for a module name.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-08-05 11:47:55 -04:00
Simon Glass
fd1b507e60 buildman: Fix use of 'boards' in test
We want to create a module called 'boards' so avoid use of this variable
name in this module. Change the global to be capitalised, as required by
Python style.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-08-05 11:47:55 -04:00
Simon Glass
938fa37c81 buildman: Fix use of 'boards' in func_test
We want to create a module called 'boards' so avoid use of this variable
name in this module. Change the global to be capitalised, as required by
Python style.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-08-05 11:47:55 -04:00
Simon Glass
f4ed4706ef buildman: Avoid using board as a variable
We have a module called 'board'. Sometimes buildman uses 'brd' as an
instance variable but sometimes it uses 'board', which is confusing and
can mess with the module handling. Update the code to use 'brd'
consistently, making it easier for tools to determine when the module
is being referenced.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-08-05 11:47:55 -04:00
Simon Glass
ae1a09f803 buildman: Support running from an IDE
Add a flag to allow buildman to behave properly for use from an IDE. This
shows error/warning output on stderr and drops all summary and progress
information.

This should normally only be used when building a single board.

Fix up a confusing comment for GetResultSummary() while we are here, since
we want to use the Outcome object to access the unprocessed error lines
from the build.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-08-05 11:47:55 -04:00
Simon Glass
5635c50720 buildman: Drop -I option
This has been deprecated with a notice that it will be removed after April
2021. Drop it now.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-08-05 11:47:55 -04:00
Alper Nebi Yasak
d8318feba1 patman: test_util: Use unittest text runner to print test results
The python tools' test utilities handle printing test results, but the
output is quite bare compared to an ordinary unittest run. Delegate
printing the results to a unittest text runner, which gives us niceties
like clear separation between each test's result and how long it took to
run the test suite.

Unfortunately it does not print info for skipped tests by default, but
this can be handled later by a custom test result subclass. It also does
not print the tool name; manually print a heading that includes the
toolname so that the outputs of each tool's tests are distinguishable in
the CI output.

Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2022-06-28 03:09:51 +01:00
Tom Rini
11232139e3 nds32: Remove the architecture
As removal of nds32 has been ack'd for the Linux kernel, remove support
here as well.

Cc: Rick Chen <rick@andestech.com>
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Rick Chen <rick@andestech.com>
2022-04-25 16:04:05 -04:00
Tom Rini
34d2b7f203 Prepare v2022.04-rc5
-----BEGIN PGP SIGNATURE-----
 
 iQGzBAABCgAdFiEEGjx/cOCPqxcHgJu/FHw5/5Y0tywFAmJBxXQACgkQFHw5/5Y0
 tywKKAv+L+sCmqfJ4Zy2cdzWkGZI3NZMWi8yzUUU37RH5I+3Blht+z6dMslm9gCB
 7/PwS5y7WH3l3wEdeXcw4FUCgNh3D8YyfCcVCsQ0Qxig49xwWz8Jn6t+haOWD6eW
 uXim+gSG+ceWTBrKf3InzXN5fHPIcF66r+k3opwAwIYKpT8p8CgM4ewshBCAsc6t
 6qSLf+A64JNv/y2DTS+sPDY68tnplbyRvrNb6YEazOF6b6SnSE2j+EH8RkIZ+2yT
 cTUhQ2o9JJIKnaG9MTqHB+XVe80oPB7bTWInFo2JxGn98XeS7CTCcnDZhLqWrvHF
 gBfcpEcVE2blVbg/Vo2x4rnU1dop44ch6pE9k5SPFJcPdlzdwFHHgGyfGi9LtD4K
 JAvaIQdz6ZEOQqShpCxcbL/g2/uoRaXFqC7E8IE+WdFMXGBQoZpqaoBMoefnyplk
 xrzqaQye38JgZDX0ku4cm4SMArAfr4sVHcdDlSlzY3y2KQ8Ead25irRvq7GoQ2Ad
 Odo9taQF
 =9g2Y
 -----END PGP SIGNATURE-----

Merge tag 'v2022.04-rc5' into next

Prepare v2022.04-rc5
2022-03-28 12:36:49 -04:00
Johannes Krottmayer
17b8cb6353 tools: buildman: Fix doc path in warning message
Fix documentation path in deprecated warning message about device
driver.

Signed-off-by: Johannes Krottmayer <krjdev@gmail.com>
Reviewed-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
2022-03-20 11:03:05 +01:00
Simon Glass
fc8af3803f buildman: Update default config to build for sandbox
At present the default .buildman file written by buildman does not specify
a default toolchain. Add an 'other' line so this works correctly and
sandbox builds run as expected.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-03-02 10:28:12 -05:00
Simon Glass
ac05335d85 buildman: Correct pylint errors
Fix pylint errors that can be fixed and mask those that seem to be
incorrect.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-03-02 10:28:12 -05:00
Simon Glass
098b10fb34 patman: Convert camel case in terminal.py
Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-02-09 12:30:13 -07:00
Simon Glass
252ac58996 patman: Rename Color() method to build()
This method has the same name as its class which is confusing. It is also
annoying when searching the code.

It builds a string with a colour, so rename it to build().

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-02-09 12:30:13 -07:00
Simon Glass
82e0e732ee patman: Rename Print() to Tprint()
Rename this function so that when we convert it to snake case it will not
conflict with the built-in print() function.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-02-09 12:30:13 -07:00
Simon Glass
5e2ab40172 patman: Convert camel case in test_util.py
Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-02-09 12:30:12 -07:00
Simon Glass
0157b187f4 patman: Convert camel case in gitutil.py
Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-02-09 12:26:12 -07:00
Simon Glass
d98006997c patman: Convert camel case in command.py
Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-02-09 12:26:12 -07:00
Simon Glass
c1aa66e75d patman: Convert camel case in tools.py
Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-02-09 12:26:12 -07:00
Simon Glass
2b4806e481 buildman: Allow adjusting board config on the fly
Add a -a option to specify changes to the config before the build
commences. For example

   buildman -a ~CONFIG_CMDLINE

disables CONFIG_CMDLINE before doing the build.

This makes it easier to try things out as well as to write tests without
creating a new board or manually manging the .config file.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-02-08 23:07:59 -05:00
Simon Glass
8ca0931a91 buildman: Provide a hint on how to debug thread crashes
If a thread crashes it is helpful to try the operation again with
threading disabled. Add a hint about that.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-02-08 23:07:59 -05:00
Simon Glass
19133b7184 buildman: Add helper functions for updating .config files
At present the only straightforward way to write tests that need a
slightly different configuration is to create a new board with its own
configuration. This is cumbersome.

It would be useful if buildman could adjust the configuration of a build
on the fly. In preparation for this, add a utility library which can
modify a .config file according to various parameters passed to it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-02-08 23:07:59 -05:00
Simon Glass
d10dc40283 buildman: Make use of test_util
Use test_util to run the tests, with the ability to select a single test
to run, if desired.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-02-08 23:07:59 -05:00
Simon Glass
433fa549e1 buildman: Add a flag to control the traceback
At present the full horror of the Python traceback is shown by default. It
is normally only useful for debugging. Turn it off by default and add a
--debug flag to enable it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-02-08 23:07:59 -05:00
Simon Glass
5b7968693f patman: Tidy up the download function a little
Reverse the order of the return tuple, so that the filename is first.
This seems more obvious than putting the temporary directory first.

Correct a bug that leaves a space on the final line.

Allow the caller to control the name of the temporary directory.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-01-25 12:36:11 -07:00
Simon Glass
8ea6d23ffb buildman: Move the download function to tools
This function is handy for binman as well. Move it into the shared 'tools'
module.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-01-25 12:36:10 -07:00
Simon Glass
c761cf778f tools: Add init files for Python tools
Add some empty __init__ files for binman, buildman and dtoc so that
pylint is able to recognise these as Python modules and produce more
useful pylint output.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-01-24 16:03:27 -05:00
Tom Rini
f7832ee552 buildman: Add gcc-11.1.0 to the directory list
While CI has been using gcc-11.1.0 for a long time, we have not updated
buildman to match.  Correct this omission.

Signed-off-by: Tom Rini <trini@konsulko.com>
2021-10-21 12:50:48 -06:00
Simon Glass
7bf83a5d7b buildman: Detect Kconfig loops
Hex and int Kconfig options are supposed to have defaults. This is so we
can configure U-Boot without having to enter particular values for the
items that don't have specific values in the board's defconfig file.

If this rule is not followed, then introducing a new Kconfig can produce
a loop like this:

   Break things (BREAK_ME) [] (NEW)
   Error in reading or end of file.

   Break things (BREAK_ME) [] (NEW)
   Error in reading or end of file.

The continues forever since buildman passes /dev/null to 'conf', and
the build system just tries again. Eventually there is so much output that
buildman runs out of memory.

We can detect this situation by looking for a symbol (like 'BREAK_ME')
which has no default (the '[]' above) and is marked as new. If this
appears multiple times in the output, we know something is wrong.

Add a filter function for the output which detects this situation. Allow
it to return True to terminate the process. Implement this termination in
cros_subprocess.

With this we get a nice message:

   buildman --board sandbox -T0
   Building current source for 1 boards (0 threads, 32 jobs per thread)
      sandbox:  w+   sandbox
   +.config:66:warning: symbol value '' invalid for BREAK_ME
   +
   +Error in reading or end of file.
   +make[3]: *** [scripts/kconfig/Makefile:75: syncconfig] Terminated
   +make[2]: *** [Makefile:569: syncconfig] Terminated
   +make: *** [Makefile:177: sub-make] Terminated
   +(** did you define an int/hex Kconfig with no default? **)

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-10-20 10:59:55 +02:00
Simon Glass
bafdeb4546 buildman: Write output even on fatal error
At present buildman does not write any output (to the 'out' and 'err)
files if the build terminates with a fatal error. This is to avoid adding
lots of spam to the logs.

However there are times when this is actually useful, such as when the
build fails for an obscure reason such as a Kconfig loop.

Update the logic to always write the output, so that the user gets a clue
as to what is happening.

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-10-20 10:59:54 +02:00
Paul Barker
5fe50f9a40 tools: Refactor full help printing
Collect the code for printing the full help message of patman, buildman
and binman into a single function in patman.tools.

Signed-off-by: Paul Barker <paul.barker@sancloud.com>
2021-09-24 14:30:46 -04:00
Simon Glass
5ed16a9511 net: Rename SPL_NET_SUPPORT to SPL_NET
Rename this option so that CONFIG_IS_ENABLED can be used with it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-09-04 12:48:53 -04:00
Simon Glass
103c5f1806 mmc: Rename MMC_SUPPORT to MMC
Rename these options so that CONFIG_IS_ENABLED can be used with them.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Jaehoon Chung <jh80.chung@samsung.com>
[trini: Fixup some incorrect renames]
Signed-off-by: Tom Rini <trini@konsulko.com>
2021-09-04 11:42:41 -04:00
Simon Glass
333e4a621d Rename SPL_USB_HOST_SUPPORT to SPL_USB_HOST
Rename this option so that CONFIG_IS_ENABLED can be used with it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-07-28 14:27:54 -04:00
Simon Glass
f1a83abe60 buildman: Use bytes for the environment
At present we sometimes see problems in gitlab where the environment has
0x80 characters or sequences which are not valid UTF-8.

Avoid this by using bytes for the environment, both internal to buildman
and when writing out the 'env' file. Add a test to make sure this works
as expected.

Reported-by: Marek Vasut <marex@denx.de>
Fixes: e5fc79ea71 ("buildman: Write the environment out to an 'env' file")
Signed-off-by: Simon Glass <sjg@chromium.org>
2021-04-29 03:23:39 -07:00
Simon Glass
8116c78ffd buildman: Handle exceptions in threads gracefully
There have been at least a few cases where an exception has occurred in a
thread and resulted in buildman hanging: running out of disk space and
getting a unicode error.

Handle these by collecting a list of exceptions, printing them out and
reporting failure if any are found. Add a test for this.

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-04-29 03:23:39 -07:00
Simon Glass
ab9b4f35e3 buildman: Use common code to send an result
At present the code to report a build result is duplicated. Put it in a
common function to avoid this.

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-04-29 03:23:39 -07:00
Simon Glass
249933136f buildman: Tidy up a few comments
Add some function comments which are missing, or missing arguments.

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-04-29 03:23:39 -07:00
Simon Glass
5f86454b3d buildman: Add an encoding to the out-env file
The environment may contain some unicode characters. At least that is what
seemed to happen on one commit:

Building current source for 1 boards (0 threads, 64 jobs per thread)
    0    0    0 /1       -1      (starting)
Traceback (most recent call last):
  File ".../tools/buildman/buildman", line 64, in <module>
    ret_code = control.DoBuildman(options, args)
  File "tools/buildman/control.py", line 372, in DoBuildman
    options.keep_outputs, options.verbose)
  File ".../tools/buildman/builder.py", line 1704, in BuildBoards
    results = self._single_builder.RunJob(job)
  File ".../tools/buildman/builderthread.py", line 526, in RunJob
    self._WriteResult(result, job.keep_outputs, job.work_in_output)
  File ".../tools//buildman/builderthread.py", line 349, in _WriteResult
    print('%s="%s"' % (var, env[var]), file=fd)
UnicodeEncodeError: 'ascii' codec can't encode characters in position
	311-312: ordinal not in range(128)

The problem defies repetition with any change at all to buildman. But
let's set an encoding in any case.

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-04-06 16:33:19 +12:00
Simon Glass
b82492bbcc buildman: Support single-threaded operation
At present even if only a single thread is in use, buildman still uses
threading.

For some debugging it is helpful to do everything in the main process.
Allow -T0 to support this.

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-03-04 17:51:43 -05:00