First CI action for performance check of LVGL code base with SO3 (#4122)

Co-authored-by: Anthony I. Jaccard <anthony.jaccard@heig-vd.ch>
Co-authored-by: AnthoJack <to.jaccard@hotmail.ch>
This commit is contained in:
Daniel Rossier 2024-06-17 22:56:51 +02:00 committed by GitHub
parent c4c66386f4
commit d9496c7979
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
12 changed files with 432 additions and 12 deletions

View File

@ -1,12 +0,0 @@
# Comment to a new issue.
pullRequestOpened: |
Thank you for raising your pull request.
To ensure that all licensing criteria is met all repositories of the LVGL project apply a process called DCO (Developer's Certificate of Origin).
The text of DCO can be read here: https://developercertificate.org/
For a more detailed description see the [Documentation](https://docs.lvgl.io/latest/en/html/contributing/index.html#developer-certification-of-origin-dco) site.
By contributing to any repositories of the LVGL project you state that your contribution corresponds with the DCO.
No further action is required if your contribution fulfills the DCO. If you are not sure about it feel free to ask us in a comment.

54
.github/workflows/check_perf.yml vendored Normal file
View File

@ -0,0 +1,54 @@
name: SO3-LVGL Performance check
on: [push, pull_request, workflow_dispatch]
jobs:
run_perf_test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
path: 'lvgl'
- name: SO3-LVGL building the Docker image
run: |
wget https://raw.githubusercontent.com/smartobjectoriented/so3/40-expand-and-document-lvgl-perf-tests/Dockerfile.lvgl
docker build . -f Dockerfile.lvgl -t so3/virt64 --platform linux/amd64 --build-arg SO3_BRANCH=40-expand-and-document-lvgl-perf-tests
- name: SO3-LVL Running containerized performance check app
run: docker run --privileged -v $PWD:/host -v /dev:/dev so3/virt64
- name: Store performance data as artifact
uses: actions/upload-artifact@v4
with:
name: performance_data
path: perf_check*.txt
- name: Find previous successful run
id: prev_success_run
run: |
wget ${{ github.api_url }}/repos/${{ github.repository }}/actions/workflows/52959381/runs
echo "ID=$(python3 lvgl/scripts/last_success_run_id.py runs)" >> "$GITHUB_OUTPUT"
- name: Retrieve previous successful run performance data
if: ${{ steps.prev_success_run.outputs.ID != 0 }}
uses: actions/download-artifact@v4
with:
name: performance_data
path: prev_performance_data
run-id: ${{ steps.prev_success_run.outputs.ID }}
continue-on-error: true
- name: Performance report generation
id: perf_report
run: |
python3 lvgl/scripts/perf_report.py perf_check*.txt lvgl/tests/perf/func_thresholds.txt prev_performance_data/perf_check*.txt | tee perf_report.txt
python_return=${PIPESTATUS[0]}
echo "Python script returned $python_return"
exit $python_return
- name: Store performance report as artifact
if: success() || failure()
uses: actions/upload-artifact@v4
with:
name: performance_report
path: perf_report.txt

Binary file not shown.

After

Width:  |  Height:  |  Size: 54 KiB

89
docs/cicd/index.rst Normal file
View File

@ -0,0 +1,89 @@
.. _cicd:
============
CICD
============
LVGL uses github actions to perform various operations on the code
Main Actions
------------
- Micropython build
- C/C++ build
- Documentation build
- PlatformIO publishing
- Release
- Performance test
LVGL Performance Test
---------------------
SO3 is used to check the performance of LVGL. This workflow behaves as shown in this diagram:
.. image:: LvglCheckPerfAction.png
The actions are described below
#. Retrieve the LVGL commit that triggered the action and store it in a "lvgl_base" folder for the dockerfile to use
#. Retrieve "Dockerfile.lvgl" from the SO3 repository and build the docker image
#. Run the docker image to generate the performance data (function execution times)
#. Store the performance data as an artifact for future reference
#. Find previous successful action run and recover the performance data from it
#. Process the performance data and compare it to previous executions and set thresholds to detect performance issues
#. Create an artifact in the form of a log file that shows the output of the regression test
The image is ran using two volumes: One that redirects the container's "/host" folder to the workflow's working directory and one that allows the container to access the workflow's devices (in the /dev folder) as his own
The workflow is setup to run when
* Commits are pushed to LVGL's repo
* A pull request is created
* Launched from another workflow
Dockerfile
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
LVGL's check_perf workflow uses the Dockerfile.lvgl found at the root of this repository to create the image that runs SO3 on qemu and executes the tests. The dockerfile does the following:
#. Creates an Alpine image and installs all the necessary tools (like gcc and qemu)
#. Recovers SO3's main branch in the root ("/") folder
#. Empties the "*/so3/usr/lib/lvgl*" folder and replaces its content with the LVGL repo to be tested (The LVGL code should be in a "lvgl_base" folder)
#. Patches SO3 so it executes the *prof_test* application instead of the shell at launch
#. Builds U-boot and the SO3 kernel
#. Imports the *prof_test* (*test/perf* folder from LVGL) application into SO3's userspace and builds the userspace
#. Sets up the image so it exposes the port 1234 when ran and executes "./st"
Performance data files
^^^^^^^^^^^^^^^^^^^^^^^
The files used to report on the execution times of the profiled functions and have the same format:
* 1 header line with no defined format (ignored by the script)
* N lines with function data following the "[parent/]<name> [(info)] | <time>" format
Parent and info are optionnal (thus marked with []).
Only one time is supported per function-parent pair
Time is expected to be a single value convertible to float. Eventual excess values will be discarded
Functions execution times are always identified by a parent-function pair (in case the function may have different behaviour depending from where it is called).Thresholds can be set for a function or a parent-function pair. parent-function thresholds are used only with exact matches in the performance data file and function thresholds are used as default for any corresponding function that does not have or does not match with a parent (for example a "main/func1" threshold is only used with "main/func1" execution times but "func2" thresholds are used for "main/func2" or "otherFunc/func2" execution times. If only "main/func1" threshold is set, no threshold is considered to be set for "otherFunc/func1" execution times)
Constraints and Guidelines
^^^^^^^^^^^^^^^^^^^^^^^^^^^
LVGL's check_perf workflow relies on a custom implementation of the *_mcount* function implemented in SO3's libc. This function is called at the start of each function found in a source file when this source file is built with GCC's "-p" (profiling) flag. The custom implementation modifies the stack to insert another function (*_mcount_exit*) as return function of the profiled function (the one that called *_mcount*). Both *_mcount* and *_mcount_exit* call a C function that timestamps their execution with the return address of the profiled function. This allows a custom script to analyse the executable file to find the function that was timestamped and calculate its execution time. This approach allows some code to be profiled automatically without the need to explicitely call timestamping functions from within the code. However it comes with a few constraints
* Build as few source files as possible with the "-p" flag. As explained, GCC will insert a call to *_mcount* at the beginning of EVERY function found in a file compiled with this flag
* Separate profiled functions from their dependencies. If a profiled function *func1* calls another function *func2*, then *func2* should be defined in a different file from *func1* and compiled without "-p" to prevent it from being profiled too. Timestamping relies on syscalls which can be slow compared to a classic execution so nesting profiled functions will result in less reliable results
The best way to profile some code is to create a new test application with 2 kinds of files:
* Setup: Those files define functions that prepare the resources necessary for the execution of the profiled functions. For example, if a profiled function needs to provide a configuration structure to one of its dependencies, the structure can be initialised and configured in it and then passed to the profiling function to be given to the function that requires it. Those files are compiled normally
* Profiling: Those files contain functions whose entry and exit time will be timestamped. They use the functions that should be profiled with the parameters given to them by the Setup functions. Those files are compiled using the "-p flag"
* The main of the application should be in a Setup file but may also be in a Profiling file if one wants to calculate the overall execution time. Please note however that whatever time is reported also measures the execution time of all the timestamping functions calls
* It is recommended to call the profiled functions from the main function as it allows the profiling data analyzer to lookup way less code to find the names of the profiled functions. This can be the difference between waiting for some seconds and waiting for minutes
Known Limitations
^^^^^^^^^^^^^^^^^^
* The current _mcount implementation is done in aarch64 assembly and is thus only compatible with ARM64 platforms
* The current _mcount implementation breaks the program if it makes use of a function with more than 7 parameters. Functions with 8 or more parameters are given some of their values through the stack in a way that is not possible to detect with our current implementation. This results in the values being shifted and replaced by the stack of _mcount_exit

View File

@ -27,6 +27,7 @@ Welcome to the documentation of LVGL!
libs/index
others/index
API/index
cicd/index
CONTRIBUTING
CODING_STYLE
CHANGELOG

17
scripts/last_success_run_id.py Executable file
View File

@ -0,0 +1,17 @@
import json
import sys
import os
if __name__ == "__main__":
if len(sys.argv) < 2 or not os.path.exists(sys.argv[1]):
print(f'Usage: {sys.argv[0]} <github_workflow_runs_json>')
exit(1)
json_file = open(sys.argv[1])
data = json.load(json_file)
for run in data["workflow_runs"]:
if run["conclusion"] == "success":
print(f'{run["id"]}')
exit(0)
print("0")

189
scripts/perf_report.py Normal file
View File

@ -0,0 +1,189 @@
"""
Compares the execution time of the functions in a file with the thresholds given in another file.
Optionnaly compares the execution times with previous executions
Prints 3 types of messages:
- FAIL: A function failed to respect the threshold set
- WARN:
- No threshold was set for a function
- No perfect match for a function-parent pair was found and no default threshold (function but no parent) was set
- A function slowed down since last time it was executed
- INFO:
- Using a default threshold for a function
- No previous execution found for a function-parent pair
"""
import os
import sys
class MalformedInputError(RuntimeError):
pass
class FuncData:
def __init__(self, name: str = "", parent: str = "", info: str = "", time: float = .0):
self.name = name # Function's name
self.parent = parent # Function parent's name
self.info = info # Optionnal supplementary data (currently unused)
self.time = time # Function's execution time
def __str__(self) -> str:
my_str = ""
if len(parent) != 0:
my_str += f'{parent}/'
my_str += f'self.name '
if len(self.info) > 0:
my_str += f'({self.info.replace(" ", "_")}) '
my_str += f'| {self.time}'
return my_str
def from_string(string: str) -> 'FuncData':
func_data = FuncData()
symbol_idx = 0
# Separate symbols of string
data = string.strip().split(" ")
try:
# Retrieve function's name and parent (if given)
func = data[symbol_idx].split("/")
if len(func) > 1:
func_data.name = func[1]
func_data.parent = func[0]
else:
func_data.name = func[0]
symbol_idx += 1
# Retrieve infos
if "(" in data[symbol_idx]:
func_data.info = data[symbol_idx].strip("()")
symbol_idx += 1
# Check that | separator is used before data
if data[symbol_idx] == "|": symbol_idx += 1
else: raise MalformedInputError("Required '|' separator is missing")
# Retrieve data
func_data.time = float(data[symbol_idx])
except IndexError:
raise MalformedInputError("Some required data is missing")
except ValueError:
raise MalformedInputError("Given data value cannot be converted to float")
except OverflowError:
raise MalformedInputError("Given data value is too big")
return func_data
"""
Input files are expected to have the following format:
1 header line with no defined format
N lines with function data following the "[parent/]<name> [(info)] | <time>" format
Parent and info are optionnal.
Only one time is supported per function-parent pair
Time is expected to be a single value convertible to float. Eventual excess values will be discarded
"""
def get_func_data_from_file(file: str) -> dict[dict[float]]:
data: dict[dict[float]] = dict()
data_file = open(file)
# Discard header line
data_file.readline()
# Parse data
for line in data_file:
# Parse line
new_data: FuncData = FuncData.from_string(line)
# Check if function already exists
if new_data.name in data.keys():
#Check that function wasn't already called from same parent
if new_data.parent in data[new_data.name].keys():
raise NotImplementedError("Current script does not support more than one execution time per parent-func pair")
else:
data[new_data.name] = dict()
#Store data
data[new_data.name][new_data.parent] = new_data.time
return data
"""
Compares the current execution time with the selected threshold
Threshold is selected based on correspondance with the execution time's function-parent pair:
- If a perfect match (function-parent) exists, select it
- If a default threshold is given for this function select it
- Otherwise, print a WARN message
"""
def check_threshold(exec_times: dict[dict[float]],
thresholds: dict[dict[float]],
func: str,
parent: str) -> bool:
if func not in thresholds.keys():
print(f'WARN: No threshold set for function {func}')
return True
# Select threshold
if parent in thresholds[func].keys():
threshold = thresholds[func][parent]
elif "" in thresholds[func].keys():
print(f'INFO: Using default threshold for function {func} called from {parent}')
threshold = thresholds[func][""]
else:
print(f'WARN: No default threshold set for function {func} called from {parent}')
return True
# Compare set threshold to calculated execution time
exec_time = exec_times[func][parent]
if exec_time > threshold:
print(f'FAIL: Function {func} called from {parent} executed in {exec_time}s but threshold is set to {threshold}s')
return False
return True
"""
Compares the current and previous execution time of the given function-parent pair
Function-parent pair must match exactely otherwise the script considers that no previous execution exists
"""
def check_prev_exec(exec_times: dict[dict[float]],
prv_exec_times: dict[dict[float]],
func: str,
parent: str):
try:
# Compare Current to previous execution
exec_time = exec_times[func][parent]
prev_exec_time = prev_exec_times[func][parent]
if exec_time > prev_exec_time:
print(f'WARN: Function {func} called from {parent} slowed from {prev_exec_time}s to {exec_time}s')
except KeyError: # No record of similar previous execution
print(f'INFO: No previous execution of function {func} called from {parent}')
return
if __name__ == "__main__":
if len(sys.argv) < 3 or len(sys.argv) > 4:
print(f'Usage: python3 {sys.argv[0]} <execution_time_file> <threshold_file> [previous_execution_time_file]')
exit(-1)
EXEC_TIME_FILE = sys.argv[1]
THRESHOLD_FILE = sys.argv[2]
PREV_EXEC_TIME_FILE = "" if len(sys.argv) < 4 else sys.argv[3]
prev_perf_exists = os.path.exists(PREV_EXEC_TIME_FILE)
if len(sys.argv) < 4 or not prev_perf_exists:
print("INFO: No previous performance data provided. Only threshold overshoot will be tested")
func_over_threshold = 0
thresholds = get_func_data_from_file(THRESHOLD_FILE)
exec_times = get_func_data_from_file(EXEC_TIME_FILE)
if prev_perf_exists:
prev_exec_times = get_func_data_from_file(PREV_EXEC_TIME_FILE)
for func in exec_times.keys():
for parent in exec_times[func].keys():
if not check_threshold(exec_times, thresholds, func, parent):
func_over_threshold += 1
continue
if prev_perf_exists:
check_prev_exec(exec_times, prev_exec_times, func, parent)
if func_over_threshold > 0:
print(f'Performance check failed: {func_over_threshold} functions over set threshold')
sys.exit(func_over_threshold)

12
tests/perf/CMakeLists.txt Normal file
View File

@ -0,0 +1,12 @@
include_directories(.)
add_executable(prof_test.elf prof_test.c)
target_compile_options(prof_test.elf PUBLIC -g)
add_library(prof_test_p.a STATIC prof_test_p.c)
target_compile_options(prof_test_p.a PRIVATE -pg)
target_link_libraries(prof_test.elf c)
target_link_libraries(prof_test.elf prof_test_p.a)
target_link_libraries(prof_test.elf lvgl)

View File

@ -0,0 +1,3 @@
[parent/]Function Name [(Optionnal_Info)] | Max execution time
prof_lv_bezier3 (success) | .001
prof_lv_area_intersect (fail) | .001

52
tests/perf/prof_test.c Normal file
View File

@ -0,0 +1,52 @@
#include "prof_test_p.h"
#include <stdio.h>
int main(int argc, char const *argv[])
{
uint32_t t = 256,
u0 = 0,
u1 = 50,
u2 = 954,
u3 = LV_BEZIER_VAL_MAX,
bezier_res1,
bezier_res2;
lv_area_t intersect_res1,
intersect_res2,
a1 = {0, 0, 4, 3},
a2 = {3, 1, 6, 4};
printf("Profiling LVGL\n");
printf("\tProfiling lv_bezier3\n");
printf("\t\tCubic bezier curve with parameters (%d;%d;%d;%d), t = %d\n", u0, u1, u2, u3, t);
bezier_res1 = lv_bezier3(t, u0, u1, u2, u3);
printf("\t\tWithout profiling: res = %d\n", bezier_res1);
bezier_res2 = prof_lv_bezier3(t, u0, u1, u2, u3);
printf("\t\tWith profiling: res = %d\n", bezier_res2);
printf("\tProfiling _lv_area_intersect\n");
printf("\t\tArea a1 (%d;%d)->(%d;%d)\n",
(int)(a1.x1),
(int)(a1.y1),
(int)(a1.x2),
(int)(a1.y2));
printf("\t\tArea a2 (%d;%d)->(%d;%d)\n",
(int)(a2.x1),
(int)(a2.y1),
(int)(a2.x2),
(int)(a2.y2));
printf("\t\tCalculating intersection\n");
_lv_area_intersect(&intersect_res1, &a1, &a2);
printf("\t\tWithout profiling: res = (%d;%d)->(%d;%d)\n",
(int)(intersect_res1.x1),
(int)(intersect_res1.y1),
(int)(intersect_res1.x2),
(int)(intersect_res1.y2));
prof_lv_area_intersect(&intersect_res2, &a1, &a2);
printf("\t\tWith profiling: res = (%d;%d)->(%d;%d)\n",
(int)(intersect_res2.x1),
(int)(intersect_res2.y1),
(int)(intersect_res2.x2),
(int)(intersect_res2.y2));
return 0;
}

11
tests/perf/prof_test_p.c Normal file
View File

@ -0,0 +1,11 @@
#include <lvgl.h>
uint32_t prof_lv_bezier3(uint32_t t, uint32_t u0, uint32_t u1, uint32_t u2, uint32_t u3)
{
return lv_bezier3(t, u0, u1, u2, u3);
}
bool prof_lv_area_intersect(lv_area_t *res_p, lv_area_t *a1_p, lv_area_t *a2_p)
{
return _lv_area_intersect(res_p, a1_p, a2_p);
}

4
tests/perf/prof_test_p.h Normal file
View File

@ -0,0 +1,4 @@
#include <lvgl.h>
uint32_t prof_lv_bezier3(uint32_t t, uint32_t u0, uint32_t u1, uint32_t u2, uint32_t u3);
bool prof_lv_area_intersect(lv_area_t *res_p, lv_area_t *a1_p, lv_area_t *a2_p);