mirror of
https://github.com/edk2-porting/linux-next.git
synced 2024-12-16 01:04:08 +08:00
Merge branch 'bpf-loader-progs'
Alexei Starovoitov says: ==================== v5->v6: - fixed issue found by bpf CI. The light skeleton generation was doing a dry-run of loading the program where all actual sys_bpf syscalls were replaced by calls into gen_loader. Turned out that search for valid vmlinux_btf was not stubbed out which was causing light skeleton gen to fail on older kernels. - significantly reduced verbosity of gen_loader.c. - an example trace_printk.lskel.h generated out of progs/trace_printk.c https://gist.github.com/4ast/774ea58f8286abac6aa8e3bf3bf3b903 v4->v5: - addressed a bunch of minor comments from Andrii. - the main difference is that lskel is now more robust in case of errors and a bit cleaner looking. v3->v4: - cleaned up closing of temporary FDs in case intermediate sys_bpf fails during execution of loader program. - added support for rodata in the skeleton. - enforce bpf_prog_type_syscall to be sleepable, since it needs bpf_copy_from_user to populate rodata map. - converted test trace_printk to use lskel to test rodata access. - various small bug fixes. v2->v3: Addressed comments from Andrii and John. - added support for setting max_entries after signature verification and used it in ringbuf test, since ringbuf's max_entries has to be updated after skeleton open() and before load(). See patch 20. - bpf_btf_find_by_name_kind doesn't take btf_fd anymore. Because of that removed attach_prog_fd from bpf_prog_desc in lskel. Both features to be added later. - cleaned up closing of fd==0 during loader gen by resetting fds back to -1. - converted loader gen to use memset(&attr, cmd_specific_attr_size). would love to see this optimization in the rest of libbpf. - fixed memory leak during loader_gen in case of enomem. - support for fd_array kernel feature is added in patch 9 to have exhaustive testing across all selftests and then partially reverted in patch 15 to keep old style map_fd patching tested as well. - since fentry_test/fexit_tests were extended with re-attach had to add support for per-program attach method in lskel and use it in the tests. - cleanup closing of fds in lskel in case of partial failures. - fixed numerous small nits. v1->v2: Addressed comments from Al, Yonghong and Andrii. - documented sys_close fdget/fdput requirement and non-recursion check. - reduced internal api leaks between libbpf and bpftool. Now bpf_object__gen_loader() is the only new libbf api with minimal fields. - fixed light skeleton __destroy() method to munmap and close maps and progs. - refactored bpf_btf_find_by_name_kind to return btf_id | (btf_obj_fd << 32). - refactored use of bpf_btf_find_by_name_kind from loader prog. - moved auto-gen like code into skel_internal.h that is used by *.lskel.h It has minimal static inline bpf_load_and_run() method used by lskel. - added lksel.h example in patch 15. - replaced union bpf_map_prog_desc with struct bpf_map_desc and struct bpf_prog_desc. - removed mark_feat_supported and added a patch to pass 'obj' into kernel_supports. - added proper tracking of temporary FDs in loader prog and their cleanup via bpf_sys_close. - rename gen_trace.c into gen_loader.c to better align the naming throughout. - expanded number of available helpers in new prog type. - added support for raw_tp attaching in lskel. lskel supports tracing and raw_tp progs now. It correctly loads all networking prog types too, but __attach() method is tbd. - converted progs/test_ksyms_module.c to lskel. - minor feedback fixes all over. The description of V1 set is still valid: This is a first step towards signed bpf programs and the third approach of that kind. The first approach was to bring libbpf into the kernel as a user-mode-driver. The second approach was to invent a new file format and let kernel execute that format as a sequence of syscalls that create maps and load programs. This third approach is using new type of bpf program instead of inventing file format. 1st and 2nd approaches had too many downsides comparing to this 3rd and were discarded after months of work. To make it work the following new concepts are introduced: 1. syscall bpf program type A kind of bpf program that can do sys_bpf and sys_close syscalls. It can only execute in user context. 2. FD array or FD index. Traditionally BPF instructions are patched with FDs. What it means that maps has to be created first and then instructions modified which breaks signature verification if the program is signed. Instead of patching each instruction with FD patch it with an index into array of FDs. That makes the program signature stable if it uses maps. 3. loader program that is generated as "strace of libbpf". When libbpf is loading bpf_file.o it does a bunch of sys_bpf() syscalls to load BTF, create maps, populate maps and finally load programs. Instead of actually doing the syscalls generate a trace of what libbpf would have done and represent it as the "loader program". The "loader program" consists of single map and single bpf program that does those syscalls. Executing such "loader program" via bpf_prog_test_run() command will replay the sequence of syscalls that libbpf would have done which will result the same maps created and programs loaded as specified in the elf file. The "loader program" removes libelf and majority of libbpf dependency from program loading process. 4. light skeleton Instead of embedding the whole elf file into skeleton and using libbpf to parse it later generate a loader program and embed it into "light skeleton". Such skeleton can load the same set of elf files, but it doesn't need libbpf and libelf to do that. It only needs few sys_bpf wrappers. Future steps: - support CO-RE in the kernel. This patch set is already too big, so that critical feature is left for the next step. - generate light skeleton in golang to allow such users use BTF and all other features provided by libbpf - generate light skeleton for kernel, so that bpf programs can be embeded in the kernel module. The UMD usage in bpf_preload will be replaced with such skeleton, so bpf_preload would become a standard kernel module without user space dependency. - finally do the signing of the loader program. The patches are work in progress with few rough edges. ==================== Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
This commit is contained in:
Daniel Borkmann
commit
3d43167768
@ -22,6 +22,7 @@
|
||||
#include <linux/sched/mm.h>
|
||||
#include <linux/slab.h>
|
||||
#include <linux/percpu-refcount.h>
|
||||
#include <linux/bpfptr.h>
|
||||
|
||||
struct bpf_verifier_env;
|
||||
struct bpf_verifier_log;
|
||||
@ -1428,7 +1429,7 @@ struct bpf_iter__bpf_map_elem {
|
||||
int bpf_iter_reg_target(const struct bpf_iter_reg *reg_info);
|
||||
void bpf_iter_unreg_target(const struct bpf_iter_reg *reg_info);
|
||||
bool bpf_iter_prog_supported(struct bpf_prog *prog);
|
||||
int bpf_iter_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
|
||||
int bpf_iter_link_attach(const union bpf_attr *attr, bpfptr_t uattr, struct bpf_prog *prog);
|
||||
int bpf_iter_new_fd(struct bpf_link *link);
|
||||
bool bpf_link_is_iter(struct bpf_link *link);
|
||||
struct bpf_prog *bpf_iter_get_info(struct bpf_iter_meta *meta, bool in_stop);
|
||||
@ -1459,7 +1460,7 @@ int bpf_fd_htab_map_update_elem(struct bpf_map *map, struct file *map_file,
|
||||
int bpf_fd_htab_map_lookup_elem(struct bpf_map *map, void *key, u32 *value);
|
||||
|
||||
int bpf_get_file_flag(int flags);
|
||||
int bpf_check_uarg_tail_zero(void __user *uaddr, size_t expected_size,
|
||||
int bpf_check_uarg_tail_zero(bpfptr_t uaddr, size_t expected_size,
|
||||
size_t actual_size);
|
||||
|
||||
/* memcpy that is used with 8-byte aligned pointers, power-of-8 size and
|
||||
@ -1479,8 +1480,7 @@ static inline void bpf_long_memcpy(void *dst, const void *src, u32 size)
|
||||
}
|
||||
|
||||
/* verify correctness of eBPF program */
|
||||
int bpf_check(struct bpf_prog **fp, union bpf_attr *attr,
|
||||
union bpf_attr __user *uattr);
|
||||
int bpf_check(struct bpf_prog **fp, union bpf_attr *attr, bpfptr_t uattr);
|
||||
|
||||
#ifndef CONFIG_BPF_JIT_ALWAYS_ON
|
||||
void bpf_patch_call_args(struct bpf_insn *insn, u32 stack_depth);
|
||||
@ -1826,6 +1826,9 @@ static inline bool bpf_map_is_dev_bound(struct bpf_map *map)
|
||||
|
||||
struct bpf_map *bpf_map_offload_map_alloc(union bpf_attr *attr);
|
||||
void bpf_map_offload_map_free(struct bpf_map *map);
|
||||
int bpf_prog_test_run_syscall(struct bpf_prog *prog,
|
||||
const union bpf_attr *kattr,
|
||||
union bpf_attr __user *uattr);
|
||||
#else
|
||||
static inline int bpf_prog_offload_init(struct bpf_prog *prog,
|
||||
union bpf_attr *attr)
|
||||
@ -1851,6 +1854,13 @@ static inline struct bpf_map *bpf_map_offload_map_alloc(union bpf_attr *attr)
|
||||
static inline void bpf_map_offload_map_free(struct bpf_map *map)
|
||||
{
|
||||
}
|
||||
|
||||
static inline int bpf_prog_test_run_syscall(struct bpf_prog *prog,
|
||||
const union bpf_attr *kattr,
|
||||
union bpf_attr __user *uattr)
|
||||
{
|
||||
return -ENOTSUPP;
|
||||
}
|
||||
#endif /* CONFIG_NET && CONFIG_BPF_SYSCALL */
|
||||
|
||||
#if defined(CONFIG_INET) && defined(CONFIG_BPF_SYSCALL)
|
||||
@ -1964,6 +1974,7 @@ extern const struct bpf_func_proto bpf_get_socket_ptr_cookie_proto;
|
||||
extern const struct bpf_func_proto bpf_task_storage_get_proto;
|
||||
extern const struct bpf_func_proto bpf_task_storage_delete_proto;
|
||||
extern const struct bpf_func_proto bpf_for_each_map_elem_proto;
|
||||
extern const struct bpf_func_proto bpf_btf_find_by_name_kind_proto;
|
||||
|
||||
const struct bpf_func_proto *bpf_tracing_func_proto(
|
||||
enum bpf_func_id func_id, const struct bpf_prog *prog);
|
||||
|
||||
@ -77,6 +77,8 @@ BPF_PROG_TYPE(BPF_PROG_TYPE_LSM, lsm,
|
||||
void *, void *)
|
||||
#endif /* CONFIG_BPF_LSM */
|
||||
#endif
|
||||
BPF_PROG_TYPE(BPF_PROG_TYPE_SYSCALL, bpf_syscall,
|
||||
void *, void *)
|
||||
|
||||
BPF_MAP_TYPE(BPF_MAP_TYPE_ARRAY, array_map_ops)
|
||||
BPF_MAP_TYPE(BPF_MAP_TYPE_PERCPU_ARRAY, percpu_array_map_ops)
|
||||
|
||||
@ -450,6 +450,7 @@ struct bpf_verifier_env {
|
||||
u32 peak_states;
|
||||
/* longest register parentage chain walked for liveness marking */
|
||||
u32 longest_mark_read_walk;
|
||||
bpfptr_t fd_array;
|
||||
};
|
||||
|
||||
__printf(2, 0) void bpf_verifier_vlog(struct bpf_verifier_log *log,
|
||||
|
||||
75
include/linux/bpfptr.h
Normal file
75
include/linux/bpfptr.h
Normal file
@ -0,0 +1,75 @@
|
||||
/* SPDX-License-Identifier: GPL-2.0-only */
|
||||
/* A pointer that can point to either kernel or userspace memory. */
|
||||
#ifndef _LINUX_BPFPTR_H
|
||||
#define _LINUX_BPFPTR_H
|
||||
|
||||
#include <linux/sockptr.h>
|
||||
|
||||
typedef sockptr_t bpfptr_t;
|
||||
|
||||
static inline bool bpfptr_is_kernel(bpfptr_t bpfptr)
|
||||
{
|
||||
return bpfptr.is_kernel;
|
||||
}
|
||||
|
||||
static inline bpfptr_t KERNEL_BPFPTR(void *p)
|
||||
{
|
||||
return (bpfptr_t) { .kernel = p, .is_kernel = true };
|
||||
}
|
||||
|
||||
static inline bpfptr_t USER_BPFPTR(void __user *p)
|
||||
{
|
||||
return (bpfptr_t) { .user = p };
|
||||
}
|
||||
|
||||
static inline bpfptr_t make_bpfptr(u64 addr, bool is_kernel)
|
||||
{
|
||||
if (is_kernel)
|
||||
return KERNEL_BPFPTR((void*) (uintptr_t) addr);
|
||||
else
|
||||
return USER_BPFPTR(u64_to_user_ptr(addr));
|
||||
}
|
||||
|
||||
static inline bool bpfptr_is_null(bpfptr_t bpfptr)
|
||||
{
|
||||
if (bpfptr_is_kernel(bpfptr))
|
||||
return !bpfptr.kernel;
|
||||
return !bpfptr.user;
|
||||
}
|
||||
|
||||
static inline void bpfptr_add(bpfptr_t *bpfptr, size_t val)
|
||||
{
|
||||
if (bpfptr_is_kernel(*bpfptr))
|
||||
bpfptr->kernel += val;
|
||||
else
|
||||
bpfptr->user += val;
|
||||
}
|
||||
|
||||
static inline int copy_from_bpfptr_offset(void *dst, bpfptr_t src,
|
||||
size_t offset, size_t size)
|
||||
{
|
||||
return copy_from_sockptr_offset(dst, (sockptr_t) src, offset, size);
|
||||
}
|
||||
|
||||
static inline int copy_from_bpfptr(void *dst, bpfptr_t src, size_t size)
|
||||
{
|
||||
return copy_from_bpfptr_offset(dst, src, 0, size);
|
||||
}
|
||||
|
||||
static inline int copy_to_bpfptr_offset(bpfptr_t dst, size_t offset,
|
||||
const void *src, size_t size)
|
||||
{
|
||||
return copy_to_sockptr_offset((sockptr_t) dst, offset, src, size);
|
||||
}
|
||||
|
||||
static inline void *memdup_bpfptr(bpfptr_t src, size_t len)
|
||||
{
|
||||
return memdup_sockptr((sockptr_t) src, len);
|
||||
}
|
||||
|
||||
static inline long strncpy_from_bpfptr(char *dst, bpfptr_t src, size_t count)
|
||||
{
|
||||
return strncpy_from_sockptr(dst, (sockptr_t) src, count);
|
||||
}
|
||||
|
||||
#endif /* _LINUX_BPFPTR_H */
|
||||
@ -21,7 +21,7 @@ extern const struct file_operations btf_fops;
|
||||
|
||||
void btf_get(struct btf *btf);
|
||||
void btf_put(struct btf *btf);
|
||||
int btf_new_fd(const union bpf_attr *attr);
|
||||
int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr);
|
||||
struct btf *btf_get_by_fd(int fd);
|
||||
int btf_get_info_by_fd(const struct btf *btf,
|
||||
const union bpf_attr *attr,
|
||||
|
||||
@ -937,6 +937,7 @@ enum bpf_prog_type {
|
||||
BPF_PROG_TYPE_EXT,
|
||||
BPF_PROG_TYPE_LSM,
|
||||
BPF_PROG_TYPE_SK_LOOKUP,
|
||||
BPF_PROG_TYPE_SYSCALL, /* a program that can execute syscalls */
|
||||
};
|
||||
|
||||
enum bpf_attach_type {
|
||||
@ -1097,8 +1098,8 @@ enum bpf_link_type {
|
||||
/* When BPF ldimm64's insn[0].src_reg != 0 then this can have
|
||||
* the following extensions:
|
||||
*
|
||||
* insn[0].src_reg: BPF_PSEUDO_MAP_FD
|
||||
* insn[0].imm: map fd
|
||||
* insn[0].src_reg: BPF_PSEUDO_MAP_[FD|IDX]
|
||||
* insn[0].imm: map fd or fd_idx
|
||||
* insn[1].imm: 0
|
||||
* insn[0].off: 0
|
||||
* insn[1].off: 0
|
||||
@ -1106,15 +1107,19 @@ enum bpf_link_type {
|
||||
* verifier type: CONST_PTR_TO_MAP
|
||||
*/
|
||||
#define BPF_PSEUDO_MAP_FD 1
|
||||
/* insn[0].src_reg: BPF_PSEUDO_MAP_VALUE
|
||||
* insn[0].imm: map fd
|
||||
#define BPF_PSEUDO_MAP_IDX 5
|
||||
|
||||
/* insn[0].src_reg: BPF_PSEUDO_MAP_[IDX_]VALUE
|
||||
* insn[0].imm: map fd or fd_idx
|
||||
* insn[1].imm: offset into value
|
||||
* insn[0].off: 0
|
||||
* insn[1].off: 0
|
||||
* ldimm64 rewrite: address of map[0]+offset
|
||||
* verifier type: PTR_TO_MAP_VALUE
|
||||
*/
|
||||
#define BPF_PSEUDO_MAP_VALUE 2
|
||||
#define BPF_PSEUDO_MAP_VALUE 2
|
||||
#define BPF_PSEUDO_MAP_IDX_VALUE 6
|
||||
|
||||
/* insn[0].src_reg: BPF_PSEUDO_BTF_ID
|
||||
* insn[0].imm: kernel btd id of VAR
|
||||
* insn[1].imm: 0
|
||||
@ -1314,6 +1319,8 @@ union bpf_attr {
|
||||
/* or valid module BTF object fd or 0 to attach to vmlinux */
|
||||
__u32 attach_btf_obj_fd;
|
||||
};
|
||||
__u32 :32; /* pad */
|
||||
__aligned_u64 fd_array; /* array of FDs */
|
||||
};
|
||||
|
||||
struct { /* anonymous struct used by BPF_OBJ_* commands */
|
||||
@ -4735,6 +4742,24 @@ union bpf_attr {
|
||||
* be zero-terminated except when **str_size** is 0.
|
||||
*
|
||||
* Or **-EBUSY** if the per-CPU memory copy buffer is busy.
|
||||
*
|
||||
* long bpf_sys_bpf(u32 cmd, void *attr, u32 attr_size)
|
||||
* Description
|
||||
* Execute bpf syscall with given arguments.
|
||||
* Return
|
||||
* A syscall result.
|
||||
*
|
||||
* long bpf_btf_find_by_name_kind(char *name, int name_sz, u32 kind, int flags)
|
||||
* Description
|
||||
* Find BTF type with given name and kind in vmlinux BTF or in module's BTFs.
|
||||
* Return
|
||||
* Returns btf_id and btf_obj_fd in lower and upper 32 bits.
|
||||
*
|
||||
* long bpf_sys_close(u32 fd)
|
||||
* Description
|
||||
* Execute close syscall for given FD.
|
||||
* Return
|
||||
* A syscall result.
|
||||
*/
|
||||
#define __BPF_FUNC_MAPPER(FN) \
|
||||
FN(unspec), \
|
||||
@ -4903,6 +4928,9 @@ union bpf_attr {
|
||||
FN(check_mtu), \
|
||||
FN(for_each_map_elem), \
|
||||
FN(snprintf), \
|
||||
FN(sys_bpf), \
|
||||
FN(btf_find_by_name_kind), \
|
||||
FN(sys_close), \
|
||||
/* */
|
||||
|
||||
/* integer value in 'imm' field of BPF_CALL instruction selects which helper
|
||||
|
||||
@ -473,15 +473,16 @@ bool bpf_link_is_iter(struct bpf_link *link)
|
||||
return link->ops == &bpf_iter_link_lops;
|
||||
}
|
||||
|
||||
int bpf_iter_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
|
||||
int bpf_iter_link_attach(const union bpf_attr *attr, bpfptr_t uattr,
|
||||
struct bpf_prog *prog)
|
||||
{
|
||||
union bpf_iter_link_info __user *ulinfo;
|
||||
struct bpf_link_primer link_primer;
|
||||
struct bpf_iter_target_info *tinfo;
|
||||
union bpf_iter_link_info linfo;
|
||||
struct bpf_iter_link *link;
|
||||
u32 prog_btf_id, linfo_len;
|
||||
bool existed = false;
|
||||
bpfptr_t ulinfo;
|
||||
int err;
|
||||
|
||||
if (attr->link_create.target_fd || attr->link_create.flags)
|
||||
@ -489,18 +490,18 @@ int bpf_iter_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
|
||||
|
||||
memset(&linfo, 0, sizeof(union bpf_iter_link_info));
|
||||
|
||||
ulinfo = u64_to_user_ptr(attr->link_create.iter_info);
|
||||
ulinfo = make_bpfptr(attr->link_create.iter_info, uattr.is_kernel);
|
||||
linfo_len = attr->link_create.iter_info_len;
|
||||
if (!ulinfo ^ !linfo_len)
|
||||
if (bpfptr_is_null(ulinfo) ^ !linfo_len)
|
||||
return -EINVAL;
|
||||
|
||||
if (ulinfo) {
|
||||
if (!bpfptr_is_null(ulinfo)) {
|
||||
err = bpf_check_uarg_tail_zero(ulinfo, sizeof(linfo),
|
||||
linfo_len);
|
||||
if (err)
|
||||
return err;
|
||||
linfo_len = min_t(u32, linfo_len, sizeof(linfo));
|
||||
if (copy_from_user(&linfo, ulinfo, linfo_len))
|
||||
if (copy_from_bpfptr(&linfo, ulinfo, linfo_len))
|
||||
return -EFAULT;
|
||||
}
|
||||
|
||||
|
||||
@ -4257,7 +4257,7 @@ static int btf_parse_hdr(struct btf_verifier_env *env)
|
||||
return 0;
|
||||
}
|
||||
|
||||
static struct btf *btf_parse(void __user *btf_data, u32 btf_data_size,
|
||||
static struct btf *btf_parse(bpfptr_t btf_data, u32 btf_data_size,
|
||||
u32 log_level, char __user *log_ubuf, u32 log_size)
|
||||
{
|
||||
struct btf_verifier_env *env = NULL;
|
||||
@ -4306,7 +4306,7 @@ static struct btf *btf_parse(void __user *btf_data, u32 btf_data_size,
|
||||
btf->data = data;
|
||||
btf->data_size = btf_data_size;
|
||||
|
||||
if (copy_from_user(data, btf_data, btf_data_size)) {
|
||||
if (copy_from_bpfptr(data, btf_data, btf_data_size)) {
|
||||
err = -EFAULT;
|
||||
goto errout;
|
||||
}
|
||||
@ -5780,12 +5780,12 @@ static int __btf_new_fd(struct btf *btf)
|
||||
return anon_inode_getfd("btf", &btf_fops, btf, O_RDONLY | O_CLOEXEC);
|
||||
}
|
||||
|
||||
int btf_new_fd(const union bpf_attr *attr)
|
||||
int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr)
|
||||
{
|
||||
struct btf *btf;
|
||||
int ret;
|
||||
|
||||
btf = btf_parse(u64_to_user_ptr(attr->btf),
|
||||
btf = btf_parse(make_bpfptr(attr->btf, uattr.is_kernel),
|
||||
attr->btf_size, attr->btf_log_level,
|
||||
u64_to_user_ptr(attr->btf_log_buf),
|
||||
attr->btf_log_size);
|
||||
@ -6085,3 +6085,65 @@ struct module *btf_try_get_module(const struct btf *btf)
|
||||
|
||||
return res;
|
||||
}
|
||||
|
||||
BPF_CALL_4(bpf_btf_find_by_name_kind, char *, name, int, name_sz, u32, kind, int, flags)
|
||||
{
|
||||
struct btf *btf;
|
||||
long ret;
|
||||
|
||||
if (flags)
|
||||
return -EINVAL;
|
||||
|
||||
if (name_sz <= 1 || name[name_sz - 1])
|
||||
return -EINVAL;
|
||||
|
||||
btf = bpf_get_btf_vmlinux();
|
||||
if (IS_ERR(btf))
|
||||
return PTR_ERR(btf);
|
||||
|
||||
ret = btf_find_by_name_kind(btf, name, kind);
|
||||
/* ret is never zero, since btf_find_by_name_kind returns
|
||||
* positive btf_id or negative error.
|
||||
*/
|
||||
if (ret < 0) {
|
||||
struct btf *mod_btf;
|
||||
int id;
|
||||
|
||||
/* If name is not found in vmlinux's BTF then search in module's BTFs */
|
||||
spin_lock_bh(&btf_idr_lock);
|
||||
idr_for_each_entry(&btf_idr, mod_btf, id) {
|
||||
if (!btf_is_module(mod_btf))
|
||||
continue;
|
||||
/* linear search could be slow hence unlock/lock
|
||||
* the IDR to avoiding holding it for too long
|
||||
*/
|
||||
btf_get(mod_btf);
|
||||
spin_unlock_bh(&btf_idr_lock);
|
||||
ret = btf_find_by_name_kind(mod_btf, name, kind);
|
||||
if (ret > 0) {
|
||||
int btf_obj_fd;
|
||||
|
||||
btf_obj_fd = __btf_new_fd(mod_btf);
|
||||
if (btf_obj_fd < 0) {
|
||||
btf_put(mod_btf);
|
||||
return btf_obj_fd;
|
||||
}
|
||||
return ret | (((u64)btf_obj_fd) << 32);
|
||||
}
|
||||
spin_lock_bh(&btf_idr_lock);
|
||||
btf_put(mod_btf);
|
||||
}
|
||||
spin_unlock_bh(&btf_idr_lock);
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
|
||||
const struct bpf_func_proto bpf_btf_find_by_name_kind_proto = {
|
||||
.func = bpf_btf_find_by_name_kind,
|
||||
.gpl_only = false,
|
||||
.ret_type = RET_INTEGER,
|
||||
.arg1_type = ARG_PTR_TO_MEM,
|
||||
.arg2_type = ARG_CONST_SIZE,
|
||||
.arg3_type = ARG_ANYTHING,
|
||||
.arg4_type = ARG_ANYTHING,
|
||||
};
|
||||
|
||||
@ -72,11 +72,10 @@ static const struct bpf_map_ops * const bpf_map_types[] = {
|
||||
* copy_from_user() call. However, this is not a concern since this function is
|
||||
* meant to be a future-proofing of bits.
|
||||
*/
|
||||
int bpf_check_uarg_tail_zero(void __user *uaddr,
|
||||
int bpf_check_uarg_tail_zero(bpfptr_t uaddr,
|
||||
size_t expected_size,
|
||||
size_t actual_size)
|
||||
{
|
||||
unsigned char __user *addr = uaddr + expected_size;
|
||||
int res;
|
||||
|
||||
if (unlikely(actual_size > PAGE_SIZE)) /* silly large */
|
||||
@ -85,7 +84,12 @@ int bpf_check_uarg_tail_zero(void __user *uaddr,
|
||||
if (actual_size <= expected_size)
|
||||
return 0;
|
||||
|
||||
res = check_zeroed_user(addr, actual_size - expected_size);
|
||||
if (uaddr.is_kernel)
|
||||
res = memchr_inv(uaddr.kernel + expected_size, 0,
|
||||
actual_size - expected_size) == NULL;
|
||||
else
|
||||
res = check_zeroed_user(uaddr.user + expected_size,
|
||||
actual_size - expected_size);
|
||||
if (res < 0)
|
||||
return res;
|
||||
return res ? 0 : -E2BIG;
|
||||
@ -1004,6 +1008,17 @@ static void *__bpf_copy_key(void __user *ukey, u64 key_size)
|
||||
return NULL;
|
||||
}
|
||||
|
||||
static void *___bpf_copy_key(bpfptr_t ukey, u64 key_size)
|
||||
{
|
||||
if (key_size)
|
||||
return memdup_bpfptr(ukey, key_size);
|
||||
|
||||
if (!bpfptr_is_null(ukey))
|
||||
return ERR_PTR(-EINVAL);
|
||||
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/* last field in 'union bpf_attr' used by this command */
|
||||
#define BPF_MAP_LOOKUP_ELEM_LAST_FIELD flags
|
||||
|
||||
@ -1074,10 +1089,10 @@ err_put:
|
||||
|
||||
#define BPF_MAP_UPDATE_ELEM_LAST_FIELD flags
|
||||
|
||||
static int map_update_elem(union bpf_attr *attr)
|
||||
static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr)
|
||||
{
|
||||
void __user *ukey = u64_to_user_ptr(attr->key);
|
||||
void __user *uvalue = u64_to_user_ptr(attr->value);
|
||||
bpfptr_t ukey = make_bpfptr(attr->key, uattr.is_kernel);
|
||||
bpfptr_t uvalue = make_bpfptr(attr->value, uattr.is_kernel);
|
||||
int ufd = attr->map_fd;
|
||||
struct bpf_map *map;
|
||||
void *key, *value;
|
||||
@ -1103,7 +1118,7 @@ static int map_update_elem(union bpf_attr *attr)
|
||||
goto err_put;
|
||||
}
|
||||
|
||||
key = __bpf_copy_key(ukey, map->key_size);
|
||||
key = ___bpf_copy_key(ukey, map->key_size);
|
||||
if (IS_ERR(key)) {
|
||||
err = PTR_ERR(key);
|
||||
goto err_put;
|
||||
@ -1123,7 +1138,7 @@ static int map_update_elem(union bpf_attr *attr)
|
||||
goto free_key;
|
||||
|
||||
err = -EFAULT;
|
||||
if (copy_from_user(value, uvalue, value_size) != 0)
|
||||
if (copy_from_bpfptr(value, uvalue, value_size) != 0)
|
||||
goto free_value;
|
||||
|
||||
err = bpf_map_update_value(map, f, key, value, attr->flags);
|
||||
@ -2014,6 +2029,7 @@ bpf_prog_load_check_attach(enum bpf_prog_type prog_type,
|
||||
if (expected_attach_type == BPF_SK_LOOKUP)
|
||||
return 0;
|
||||
return -EINVAL;
|
||||
case BPF_PROG_TYPE_SYSCALL:
|
||||
case BPF_PROG_TYPE_EXT:
|
||||
if (expected_attach_type)
|
||||
return -EINVAL;
|
||||
@ -2073,9 +2089,9 @@ static bool is_perfmon_prog_type(enum bpf_prog_type prog_type)
|
||||
}
|
||||
|
||||
/* last field in 'union bpf_attr' used by this command */
|
||||
#define BPF_PROG_LOAD_LAST_FIELD attach_prog_fd
|
||||
#define BPF_PROG_LOAD_LAST_FIELD fd_array
|
||||
|
||||
static int bpf_prog_load(union bpf_attr *attr, union bpf_attr __user *uattr)
|
||||
static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr)
|
||||
{
|
||||
enum bpf_prog_type type = attr->prog_type;
|
||||
struct bpf_prog *prog, *dst_prog = NULL;
|
||||
@ -2100,8 +2116,9 @@ static int bpf_prog_load(union bpf_attr *attr, union bpf_attr __user *uattr)
|
||||
return -EPERM;
|
||||
|
||||
/* copy eBPF program license from user space */
|
||||
if (strncpy_from_user(license, u64_to_user_ptr(attr->license),
|
||||
sizeof(license) - 1) < 0)
|
||||
if (strncpy_from_bpfptr(license,
|
||||
make_bpfptr(attr->license, uattr.is_kernel),
|
||||
sizeof(license) - 1) < 0)
|
||||
return -EFAULT;
|
||||
license[sizeof(license) - 1] = 0;
|
||||
|
||||
@ -2185,8 +2202,9 @@ static int bpf_prog_load(union bpf_attr *attr, union bpf_attr __user *uattr)
|
||||
prog->len = attr->insn_cnt;
|
||||
|
||||
err = -EFAULT;
|
||||
if (copy_from_user(prog->insns, u64_to_user_ptr(attr->insns),
|
||||
bpf_prog_insn_size(prog)) != 0)
|
||||
if (copy_from_bpfptr(prog->insns,
|
||||
make_bpfptr(attr->insns, uattr.is_kernel),
|
||||
bpf_prog_insn_size(prog)) != 0)
|
||||
goto free_prog_sec;
|
||||
|
||||
prog->orig_prog = NULL;
|
||||
@ -3422,7 +3440,7 @@ static int bpf_prog_get_info_by_fd(struct file *file,
|
||||
u32 ulen;
|
||||
int err;
|
||||
|
||||
err = bpf_check_uarg_tail_zero(uinfo, sizeof(info), info_len);
|
||||
err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), sizeof(info), info_len);
|
||||
if (err)
|
||||
return err;
|
||||
info_len = min_t(u32, sizeof(info), info_len);
|
||||
@ -3701,7 +3719,7 @@ static int bpf_map_get_info_by_fd(struct file *file,
|
||||
u32 info_len = attr->info.info_len;
|
||||
int err;
|
||||
|
||||
err = bpf_check_uarg_tail_zero(uinfo, sizeof(info), info_len);
|
||||
err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), sizeof(info), info_len);
|
||||
if (err)
|
||||
return err;
|
||||
info_len = min_t(u32, sizeof(info), info_len);
|
||||
@ -3744,7 +3762,7 @@ static int bpf_btf_get_info_by_fd(struct file *file,
|
||||
u32 info_len = attr->info.info_len;
|
||||
int err;
|
||||
|
||||
err = bpf_check_uarg_tail_zero(uinfo, sizeof(*uinfo), info_len);
|
||||
err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), sizeof(*uinfo), info_len);
|
||||
if (err)
|
||||
return err;
|
||||
|
||||
@ -3761,7 +3779,7 @@ static int bpf_link_get_info_by_fd(struct file *file,
|
||||
u32 info_len = attr->info.info_len;
|
||||
int err;
|
||||
|
||||
err = bpf_check_uarg_tail_zero(uinfo, sizeof(info), info_len);
|
||||
err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), sizeof(info), info_len);
|
||||
if (err)
|
||||
return err;
|
||||
info_len = min_t(u32, sizeof(info), info_len);
|
||||
@ -3824,7 +3842,7 @@ static int bpf_obj_get_info_by_fd(const union bpf_attr *attr,
|
||||
|
||||
#define BPF_BTF_LOAD_LAST_FIELD btf_log_level
|
||||
|
||||
static int bpf_btf_load(const union bpf_attr *attr)
|
||||
static int bpf_btf_load(const union bpf_attr *attr, bpfptr_t uattr)
|
||||
{
|
||||
if (CHECK_ATTR(BPF_BTF_LOAD))
|
||||
return -EINVAL;
|
||||
@ -3832,7 +3850,7 @@ static int bpf_btf_load(const union bpf_attr *attr)
|
||||
if (!bpf_capable())
|
||||
return -EPERM;
|
||||
|
||||
return btf_new_fd(attr);
|
||||
return btf_new_fd(attr, uattr);
|
||||
}
|
||||
|
||||
#define BPF_BTF_GET_FD_BY_ID_LAST_FIELD btf_id
|
||||
@ -4022,13 +4040,14 @@ err_put:
|
||||
return err;
|
||||
}
|
||||
|
||||
static int tracing_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
|
||||
static int tracing_bpf_link_attach(const union bpf_attr *attr, bpfptr_t uattr,
|
||||
struct bpf_prog *prog)
|
||||
{
|
||||
if (attr->link_create.attach_type != prog->expected_attach_type)
|
||||
return -EINVAL;
|
||||
|
||||
if (prog->expected_attach_type == BPF_TRACE_ITER)
|
||||
return bpf_iter_link_attach(attr, prog);
|
||||
return bpf_iter_link_attach(attr, uattr, prog);
|
||||
else if (prog->type == BPF_PROG_TYPE_EXT)
|
||||
return bpf_tracing_prog_attach(prog,
|
||||
attr->link_create.target_fd,
|
||||
@ -4037,7 +4056,7 @@ static int tracing_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *
|
||||
}
|
||||
|
||||
#define BPF_LINK_CREATE_LAST_FIELD link_create.iter_info_len
|
||||
static int link_create(union bpf_attr *attr)
|
||||
static int link_create(union bpf_attr *attr, bpfptr_t uattr)
|
||||
{
|
||||
enum bpf_prog_type ptype;
|
||||
struct bpf_prog *prog;
|
||||
@ -4056,7 +4075,7 @@ static int link_create(union bpf_attr *attr)
|
||||
goto out;
|
||||
|
||||
if (prog->type == BPF_PROG_TYPE_EXT) {
|
||||
ret = tracing_bpf_link_attach(attr, prog);
|
||||
ret = tracing_bpf_link_attach(attr, uattr, prog);
|
||||
goto out;
|
||||
}
|
||||
|
||||
@ -4077,7 +4096,7 @@ static int link_create(union bpf_attr *attr)
|
||||
ret = cgroup_bpf_link_attach(attr, prog);
|
||||
break;
|
||||
case BPF_PROG_TYPE_TRACING:
|
||||
ret = tracing_bpf_link_attach(attr, prog);
|
||||
ret = tracing_bpf_link_attach(attr, uattr, prog);
|
||||
break;
|
||||
case BPF_PROG_TYPE_FLOW_DISSECTOR:
|
||||
case BPF_PROG_TYPE_SK_LOOKUP:
|
||||
@ -4365,7 +4384,7 @@ out_prog_put:
|
||||
return ret;
|
||||
}
|
||||
|
||||
SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size)
|
||||
static int __sys_bpf(int cmd, bpfptr_t uattr, unsigned int size)
|
||||
{
|
||||
union bpf_attr attr;
|
||||
int err;
|
||||
@ -4380,7 +4399,7 @@ SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, siz
|
||||
|
||||
/* copy attributes from user space, may be less than sizeof(bpf_attr) */
|
||||
memset(&attr, 0, sizeof(attr));
|
||||
if (copy_from_user(&attr, uattr, size) != 0)
|
||||
if (copy_from_bpfptr(&attr, uattr, size) != 0)
|
||||
return -EFAULT;
|
||||
|
||||
err = security_bpf(cmd, &attr, size);
|
||||
@ -4395,7 +4414,7 @@ SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, siz
|
||||
err = map_lookup_elem(&attr);
|
||||
break;
|
||||
case BPF_MAP_UPDATE_ELEM:
|
||||
err = map_update_elem(&attr);
|
||||
err = map_update_elem(&attr, uattr);
|
||||
break;
|
||||
case BPF_MAP_DELETE_ELEM:
|
||||
err = map_delete_elem(&attr);
|
||||
@ -4422,21 +4441,21 @@ SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, siz
|
||||
err = bpf_prog_detach(&attr);
|
||||
break;
|
||||
case BPF_PROG_QUERY:
|
||||
err = bpf_prog_query(&attr, uattr);
|
||||
err = bpf_prog_query(&attr, uattr.user);
|
||||
break;
|
||||
case BPF_PROG_TEST_RUN:
|
||||
err = bpf_prog_test_run(&attr, uattr);
|
||||
err = bpf_prog_test_run(&attr, uattr.user);
|
||||
break;
|
||||
case BPF_PROG_GET_NEXT_ID:
|
||||
err = bpf_obj_get_next_id(&attr, uattr,
|
||||
err = bpf_obj_get_next_id(&attr, uattr.user,
|
||||
&prog_idr, &prog_idr_lock);
|
||||
break;
|
||||
case BPF_MAP_GET_NEXT_ID:
|
||||
err = bpf_obj_get_next_id(&attr, uattr,
|
||||
err = bpf_obj_get_next_id(&attr, uattr.user,
|
||||
&map_idr, &map_idr_lock);
|
||||
break;
|
||||
case BPF_BTF_GET_NEXT_ID:
|
||||
err = bpf_obj_get_next_id(&attr, uattr,
|
||||
err = bpf_obj_get_next_id(&attr, uattr.user,
|
||||
&btf_idr, &btf_idr_lock);
|
||||
break;
|
||||
case BPF_PROG_GET_FD_BY_ID:
|
||||
@ -4446,38 +4465,38 @@ SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, siz
|
||||
err = bpf_map_get_fd_by_id(&attr);
|
||||
break;
|
||||
case BPF_OBJ_GET_INFO_BY_FD:
|
||||
err = bpf_obj_get_info_by_fd(&attr, uattr);
|
||||
err = bpf_obj_get_info_by_fd(&attr, uattr.user);
|
||||
break;
|
||||
case BPF_RAW_TRACEPOINT_OPEN:
|
||||
err = bpf_raw_tracepoint_open(&attr);
|
||||
break;
|
||||
case BPF_BTF_LOAD:
|
||||
err = bpf_btf_load(&attr);
|
||||
err = bpf_btf_load(&attr, uattr);
|
||||
break;
|
||||
case BPF_BTF_GET_FD_BY_ID:
|
||||
err = bpf_btf_get_fd_by_id(&attr);
|
||||
break;
|
||||
case BPF_TASK_FD_QUERY:
|
||||
err = bpf_task_fd_query(&attr, uattr);
|
||||
err = bpf_task_fd_query(&attr, uattr.user);
|
||||
break;
|
||||
case BPF_MAP_LOOKUP_AND_DELETE_ELEM:
|
||||
err = map_lookup_and_delete_elem(&attr);
|
||||
break;
|
||||
case BPF_MAP_LOOKUP_BATCH:
|
||||
err = bpf_map_do_batch(&attr, uattr, BPF_MAP_LOOKUP_BATCH);
|
||||
err = bpf_map_do_batch(&attr, uattr.user, BPF_MAP_LOOKUP_BATCH);
|
||||
break;
|
||||
case BPF_MAP_LOOKUP_AND_DELETE_BATCH:
|
||||
err = bpf_map_do_batch(&attr, uattr,
|
||||
err = bpf_map_do_batch(&attr, uattr.user,
|
||||
BPF_MAP_LOOKUP_AND_DELETE_BATCH);
|
||||
break;
|
||||
case BPF_MAP_UPDATE_BATCH:
|
||||
err = bpf_map_do_batch(&attr, uattr, BPF_MAP_UPDATE_BATCH);
|
||||
err = bpf_map_do_batch(&attr, uattr.user, BPF_MAP_UPDATE_BATCH);
|
||||
break;
|
||||
case BPF_MAP_DELETE_BATCH:
|
||||
err = bpf_map_do_batch(&attr, uattr, BPF_MAP_DELETE_BATCH);
|
||||
err = bpf_map_do_batch(&attr, uattr.user, BPF_MAP_DELETE_BATCH);
|
||||
break;
|
||||
case BPF_LINK_CREATE:
|
||||
err = link_create(&attr);
|
||||
err = link_create(&attr, uattr);
|
||||
break;
|
||||
case BPF_LINK_UPDATE:
|
||||
err = link_update(&attr);
|
||||
@ -4486,7 +4505,7 @@ SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, siz
|
||||
err = bpf_link_get_fd_by_id(&attr);
|
||||
break;
|
||||
case BPF_LINK_GET_NEXT_ID:
|
||||
err = bpf_obj_get_next_id(&attr, uattr,
|
||||
err = bpf_obj_get_next_id(&attr, uattr.user,
|
||||
&link_idr, &link_idr_lock);
|
||||
break;
|
||||
case BPF_ENABLE_STATS:
|
||||
@ -4508,3 +4527,94 @@ SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, siz
|
||||
|
||||
return err;
|
||||
}
|
||||
|
||||
SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size)
|
||||
{
|
||||
return __sys_bpf(cmd, USER_BPFPTR(uattr), size);
|
||||
}
|
||||
|
||||
static bool syscall_prog_is_valid_access(int off, int size,
|
||||
enum bpf_access_type type,
|
||||
const struct bpf_prog *prog,
|
||||
struct bpf_insn_access_aux *info)
|
||||
{
|
||||
if (off < 0 || off >= U16_MAX)
|
||||
return false;
|
||||
if (off % size != 0)
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
BPF_CALL_3(bpf_sys_bpf, int, cmd, void *, attr, u32, attr_size)
|
||||
{
|
||||
switch (cmd) {
|
||||
case BPF_MAP_CREATE:
|
||||
case BPF_MAP_UPDATE_ELEM:
|
||||
case BPF_MAP_FREEZE:
|
||||
case BPF_PROG_LOAD:
|
||||
case BPF_BTF_LOAD:
|
||||
break;
|
||||
/* case BPF_PROG_TEST_RUN:
|
||||
* is not part of this list to prevent recursive test_run
|
||||
*/
|
||||
default:
|
||||
return -EINVAL;
|
||||
}
|
||||
return __sys_bpf(cmd, KERNEL_BPFPTR(attr), attr_size);
|
||||
}
|
||||
|
||||
const struct bpf_func_proto bpf_sys_bpf_proto = {
|
||||
.func = bpf_sys_bpf,
|
||||
.gpl_only = false,
|
||||
.ret_type = RET_INTEGER,
|
||||
.arg1_type = ARG_ANYTHING,
|
||||
.arg2_type = ARG_PTR_TO_MEM,
|
||||
.arg3_type = ARG_CONST_SIZE,
|
||||
};
|
||||
|
||||
const struct bpf_func_proto * __weak
|
||||
tracing_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
|
||||
{
|
||||
return bpf_base_func_proto(func_id);
|
||||
}
|
||||
|
||||
BPF_CALL_1(bpf_sys_close, u32, fd)
|
||||
{
|
||||
/* When bpf program calls this helper there should not be
|
||||
* an fdget() without matching completed fdput().
|
||||
* This helper is allowed in the following callchain only:
|
||||
* sys_bpf->prog_test_run->bpf_prog->bpf_sys_close
|
||||
*/
|
||||
return close_fd(fd);
|
||||
}
|
||||
|
||||
const struct bpf_func_proto bpf_sys_close_proto = {
|
||||
.func = bpf_sys_close,
|
||||
.gpl_only = false,
|
||||
.ret_type = RET_INTEGER,
|
||||
.arg1_type = ARG_ANYTHING,
|
||||
};
|
||||
|
||||
static const struct bpf_func_proto *
|
||||
syscall_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
|
||||
{
|
||||
switch (func_id) {
|
||||
case BPF_FUNC_sys_bpf:
|
||||
return &bpf_sys_bpf_proto;
|
||||
case BPF_FUNC_btf_find_by_name_kind:
|
||||
return &bpf_btf_find_by_name_kind_proto;
|
||||
case BPF_FUNC_sys_close:
|
||||
return &bpf_sys_close_proto;
|
||||
default:
|
||||
return tracing_prog_func_proto(func_id, prog);
|
||||
}
|
||||
}
|
||||
|
||||
const struct bpf_verifier_ops bpf_syscall_verifier_ops = {
|
||||
.get_func_proto = syscall_prog_func_proto,
|
||||
.is_valid_access = syscall_prog_is_valid_access,
|
||||
};
|
||||
|
||||
const struct bpf_prog_ops bpf_syscall_prog_ops = {
|
||||
.test_run = bpf_prog_test_run_syscall,
|
||||
};
|
||||
|
||||
@ -8915,12 +8915,14 @@ static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn)
|
||||
mark_reg_known_zero(env, regs, insn->dst_reg);
|
||||
dst_reg->map_ptr = map;
|
||||
|
||||
if (insn->src_reg == BPF_PSEUDO_MAP_VALUE) {
|
||||
if (insn->src_reg == BPF_PSEUDO_MAP_VALUE ||
|
||||
insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE) {
|
||||
dst_reg->type = PTR_TO_MAP_VALUE;
|
||||
dst_reg->off = aux->map_off;
|
||||
if (map_value_has_spin_lock(map))
|
||||
dst_reg->id = ++env->id_gen;
|
||||
} else if (insn->src_reg == BPF_PSEUDO_MAP_FD) {
|
||||
} else if (insn->src_reg == BPF_PSEUDO_MAP_FD ||
|
||||
insn->src_reg == BPF_PSEUDO_MAP_IDX) {
|
||||
dst_reg->type = CONST_PTR_TO_MAP;
|
||||
} else {
|
||||
verbose(env, "bpf verifier is misconfigured\n");
|
||||
@ -9436,7 +9438,7 @@ static int check_abnormal_return(struct bpf_verifier_env *env)
|
||||
|
||||
static int check_btf_func(struct bpf_verifier_env *env,
|
||||
const union bpf_attr *attr,
|
||||
union bpf_attr __user *uattr)
|
||||
bpfptr_t uattr)
|
||||
{
|
||||
const struct btf_type *type, *func_proto, *ret_type;
|
||||
u32 i, nfuncs, urec_size, min_size;
|
||||
@ -9445,7 +9447,7 @@ static int check_btf_func(struct bpf_verifier_env *env,
|
||||
struct bpf_func_info_aux *info_aux = NULL;
|
||||
struct bpf_prog *prog;
|
||||
const struct btf *btf;
|
||||
void __user *urecord;
|
||||
bpfptr_t urecord;
|
||||
u32 prev_offset = 0;
|
||||
bool scalar_return;
|
||||
int ret = -ENOMEM;
|
||||
@ -9473,7 +9475,7 @@ static int check_btf_func(struct bpf_verifier_env *env,
|
||||
prog = env->prog;
|
||||
btf = prog->aux->btf;
|
||||
|
||||
urecord = u64_to_user_ptr(attr->func_info);
|
||||
urecord = make_bpfptr(attr->func_info, uattr.is_kernel);
|
||||
min_size = min_t(u32, krec_size, urec_size);
|
||||
|
||||
krecord = kvcalloc(nfuncs, krec_size, GFP_KERNEL | __GFP_NOWARN);
|
||||
@ -9491,13 +9493,15 @@ static int check_btf_func(struct bpf_verifier_env *env,
|
||||
/* set the size kernel expects so loader can zero
|
||||
* out the rest of the record.
|
||||
*/
|
||||
if (put_user(min_size, &uattr->func_info_rec_size))
|
||||
if (copy_to_bpfptr_offset(uattr,
|
||||
offsetof(union bpf_attr, func_info_rec_size),
|
||||
&min_size, sizeof(min_size)))
|
||||
ret = -EFAULT;
|
||||
}
|
||||
goto err_free;
|
||||
}
|
||||
|
||||
if (copy_from_user(&krecord[i], urecord, min_size)) {
|
||||
if (copy_from_bpfptr(&krecord[i], urecord, min_size)) {
|
||||
ret = -EFAULT;
|
||||
goto err_free;
|
||||
}
|
||||
@ -9549,7 +9553,7 @@ static int check_btf_func(struct bpf_verifier_env *env,
|
||||
}
|
||||
|
||||
prev_offset = krecord[i].insn_off;
|
||||
urecord += urec_size;
|
||||
bpfptr_add(&urecord, urec_size);
|
||||
}
|
||||
|
||||
prog->aux->func_info = krecord;
|
||||
@ -9581,14 +9585,14 @@ static void adjust_btf_func(struct bpf_verifier_env *env)
|
||||
|
||||
static int check_btf_line(struct bpf_verifier_env *env,
|
||||
const union bpf_attr *attr,
|
||||
union bpf_attr __user *uattr)
|
||||
bpfptr_t uattr)
|
||||
{
|
||||
u32 i, s, nr_linfo, ncopy, expected_size, rec_size, prev_offset = 0;
|
||||
struct bpf_subprog_info *sub;
|
||||
struct bpf_line_info *linfo;
|
||||
struct bpf_prog *prog;
|
||||
const struct btf *btf;
|
||||
void __user *ulinfo;
|
||||
bpfptr_t ulinfo;
|
||||
int err;
|
||||
|
||||
nr_linfo = attr->line_info_cnt;
|
||||
@ -9614,7 +9618,7 @@ static int check_btf_line(struct bpf_verifier_env *env,
|
||||
|
||||
s = 0;
|
||||
sub = env->subprog_info;
|
||||
ulinfo = u64_to_user_ptr(attr->line_info);
|
||||
ulinfo = make_bpfptr(attr->line_info, uattr.is_kernel);
|
||||
expected_size = sizeof(struct bpf_line_info);
|
||||
ncopy = min_t(u32, expected_size, rec_size);
|
||||
for (i = 0; i < nr_linfo; i++) {
|
||||
@ -9622,14 +9626,15 @@ static int check_btf_line(struct bpf_verifier_env *env,
|
||||
if (err) {
|
||||
if (err == -E2BIG) {
|
||||
verbose(env, "nonzero tailing record in line_info");
|
||||
if (put_user(expected_size,
|
||||
&uattr->line_info_rec_size))
|
||||
if (copy_to_bpfptr_offset(uattr,
|
||||
offsetof(union bpf_attr, line_info_rec_size),
|
||||
&expected_size, sizeof(expected_size)))
|
||||
err = -EFAULT;
|
||||
}
|
||||
goto err_free;
|
||||
}
|
||||
|
||||
if (copy_from_user(&linfo[i], ulinfo, ncopy)) {
|
||||
if (copy_from_bpfptr(&linfo[i], ulinfo, ncopy)) {
|
||||
err = -EFAULT;
|
||||
goto err_free;
|
||||
}
|
||||
@ -9681,7 +9686,7 @@ static int check_btf_line(struct bpf_verifier_env *env,
|
||||
}
|
||||
|
||||
prev_offset = linfo[i].insn_off;
|
||||
ulinfo += rec_size;
|
||||
bpfptr_add(&ulinfo, rec_size);
|
||||
}
|
||||
|
||||
if (s != env->subprog_cnt) {
|
||||
@ -9703,7 +9708,7 @@ err_free:
|
||||
|
||||
static int check_btf_info(struct bpf_verifier_env *env,
|
||||
const union bpf_attr *attr,
|
||||
union bpf_attr __user *uattr)
|
||||
bpfptr_t uattr)
|
||||
{
|
||||
struct btf *btf;
|
||||
int err;
|
||||
@ -11170,6 +11175,7 @@ static int resolve_pseudo_ldimm64(struct bpf_verifier_env *env)
|
||||
struct bpf_map *map;
|
||||
struct fd f;
|
||||
u64 addr;
|
||||
u32 fd;
|
||||
|
||||
if (i == insn_cnt - 1 || insn[1].code != 0 ||
|
||||
insn[1].dst_reg != 0 || insn[1].src_reg != 0 ||
|
||||
@ -11199,16 +11205,38 @@ static int resolve_pseudo_ldimm64(struct bpf_verifier_env *env)
|
||||
/* In final convert_pseudo_ld_imm64() step, this is
|
||||
* converted into regular 64-bit imm load insn.
|
||||
*/
|
||||
if ((insn[0].src_reg != BPF_PSEUDO_MAP_FD &&
|
||||
insn[0].src_reg != BPF_PSEUDO_MAP_VALUE) ||
|
||||
(insn[0].src_reg == BPF_PSEUDO_MAP_FD &&
|
||||
insn[1].imm != 0)) {
|
||||
verbose(env,
|
||||
"unrecognized bpf_ld_imm64 insn\n");
|
||||
switch (insn[0].src_reg) {
|
||||
case BPF_PSEUDO_MAP_VALUE:
|
||||
case BPF_PSEUDO_MAP_IDX_VALUE:
|
||||
break;
|
||||
case BPF_PSEUDO_MAP_FD:
|
||||
case BPF_PSEUDO_MAP_IDX:
|
||||
if (insn[1].imm == 0)
|
||||
break;
|
||||
fallthrough;
|
||||
default:
|
||||
verbose(env, "unrecognized bpf_ld_imm64 insn\n");
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
f = fdget(insn[0].imm);
|
||||
switch (insn[0].src_reg) {
|
||||
case BPF_PSEUDO_MAP_IDX_VALUE:
|
||||
case BPF_PSEUDO_MAP_IDX:
|
||||
if (bpfptr_is_null(env->fd_array)) {
|
||||
verbose(env, "fd_idx without fd_array is invalid\n");
|
||||
return -EPROTO;
|
||||
}
|
||||
if (copy_from_bpfptr_offset(&fd, env->fd_array,
|
||||
insn[0].imm * sizeof(fd),
|
||||
sizeof(fd)))
|
||||
return -EFAULT;
|
||||
break;
|
||||
default:
|
||||
fd = insn[0].imm;
|
||||
break;
|
||||
}
|
||||
|
||||
f = fdget(fd);
|
||||
map = __bpf_map_get(f);
|
||||
if (IS_ERR(map)) {
|
||||
verbose(env, "fd %d is not pointing to valid bpf_map\n",
|
||||
@ -11223,7 +11251,8 @@ static int resolve_pseudo_ldimm64(struct bpf_verifier_env *env)
|
||||
}
|
||||
|
||||
aux = &env->insn_aux_data[i];
|
||||
if (insn->src_reg == BPF_PSEUDO_MAP_FD) {
|
||||
if (insn[0].src_reg == BPF_PSEUDO_MAP_FD ||
|
||||
insn[0].src_reg == BPF_PSEUDO_MAP_IDX) {
|
||||
addr = (unsigned long)map;
|
||||
} else {
|
||||
u32 off = insn[1].imm;
|
||||
@ -13196,6 +13225,14 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)
|
||||
int ret;
|
||||
u64 key;
|
||||
|
||||
if (prog->type == BPF_PROG_TYPE_SYSCALL) {
|
||||
if (prog->aux->sleepable)
|
||||
/* attach_btf_id checked to be zero already */
|
||||
return 0;
|
||||
verbose(env, "Syscall programs can only be sleepable\n");
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
if (prog->aux->sleepable && prog->type != BPF_PROG_TYPE_TRACING &&
|
||||
prog->type != BPF_PROG_TYPE_LSM) {
|
||||
verbose(env, "Only fentry/fexit/fmod_ret and lsm programs can be sleepable\n");
|
||||
@ -13267,8 +13304,7 @@ struct btf *bpf_get_btf_vmlinux(void)
|
||||
return btf_vmlinux;
|
||||
}
|
||||
|
||||
int bpf_check(struct bpf_prog **prog, union bpf_attr *attr,
|
||||
union bpf_attr __user *uattr)
|
||||
int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr)
|
||||
{
|
||||
u64 start_time = ktime_get_ns();
|
||||
struct bpf_verifier_env *env;
|
||||
@ -13298,6 +13334,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr,
|
||||
env->insn_aux_data[i].orig_idx = i;
|
||||
env->prog = *prog;
|
||||
env->ops = bpf_verifier_ops[env->prog->type];
|
||||
env->fd_array = make_bpfptr(attr->fd_array, uattr.is_kernel);
|
||||
is_priv = bpf_capable();
|
||||
|
||||
bpf_get_btf_vmlinux();
|
||||
|
||||
@ -409,7 +409,7 @@ static void *bpf_ctx_init(const union bpf_attr *kattr, u32 max_size)
|
||||
return ERR_PTR(-ENOMEM);
|
||||
|
||||
if (data_in) {
|
||||
err = bpf_check_uarg_tail_zero(data_in, max_size, size);
|
||||
err = bpf_check_uarg_tail_zero(USER_BPFPTR(data_in), max_size, size);
|
||||
if (err) {
|
||||
kfree(data);
|
||||
return ERR_PTR(err);
|
||||
@ -918,3 +918,46 @@ out:
|
||||
kfree(user_ctx);
|
||||
return ret;
|
||||
}
|
||||
|
||||
int bpf_prog_test_run_syscall(struct bpf_prog *prog,
|
||||
const union bpf_attr *kattr,
|
||||
union bpf_attr __user *uattr)
|
||||
{
|
||||
void __user *ctx_in = u64_to_user_ptr(kattr->test.ctx_in);
|
||||
__u32 ctx_size_in = kattr->test.ctx_size_in;
|
||||
void *ctx = NULL;
|
||||
u32 retval;
|
||||
int err = 0;
|
||||
|
||||
/* doesn't support data_in/out, ctx_out, duration, or repeat or flags */
|
||||
if (kattr->test.data_in || kattr->test.data_out ||
|
||||
kattr->test.ctx_out || kattr->test.duration ||
|
||||
kattr->test.repeat || kattr->test.flags)
|
||||
return -EINVAL;
|
||||
|
||||
if (ctx_size_in < prog->aux->max_ctx_offset ||
|
||||
ctx_size_in > U16_MAX)
|
||||
return -EINVAL;
|
||||
|
||||
if (ctx_size_in) {
|
||||
ctx = kzalloc(ctx_size_in, GFP_USER);
|
||||
if (!ctx)
|
||||
return -ENOMEM;
|
||||
if (copy_from_user(ctx, ctx_in, ctx_size_in)) {
|
||||
err = -EFAULT;
|
||||
goto out;
|
||||
}
|
||||
}
|
||||
retval = bpf_prog_run_pin_on_cpu(prog, ctx);
|
||||
|
||||
if (copy_to_user(&uattr->test.retval, &retval, sizeof(u32))) {
|
||||
err = -EFAULT;
|
||||
goto out;
|
||||
}
|
||||
if (ctx_size_in)
|
||||
if (copy_to_user(ctx_in, ctx, ctx_size_in))
|
||||
err = -EFAULT;
|
||||
out:
|
||||
kfree(ctx);
|
||||
return err;
|
||||
}
|
||||
|
||||
@ -136,7 +136,7 @@ endif
|
||||
|
||||
BPFTOOL_BOOTSTRAP := $(BOOTSTRAP_OUTPUT)bpftool
|
||||
|
||||
BOOTSTRAP_OBJS = $(addprefix $(BOOTSTRAP_OUTPUT),main.o common.o json_writer.o gen.o btf.o)
|
||||
BOOTSTRAP_OBJS = $(addprefix $(BOOTSTRAP_OUTPUT),main.o common.o json_writer.o gen.o btf.o xlated_dumper.o btf_dumper.o) $(OUTPUT)disasm.o
|
||||
OBJS = $(patsubst %.c,$(OUTPUT)%.o,$(SRCS)) $(OUTPUT)disasm.o
|
||||
|
||||
VMLINUX_BTF_PATHS ?= $(if $(O),$(O)/vmlinux) \
|
||||
|
||||
@ -18,6 +18,7 @@
|
||||
#include <sys/stat.h>
|
||||
#include <sys/mman.h>
|
||||
#include <bpf/btf.h>
|
||||
#include <bpf/bpf_gen_internal.h>
|
||||
|
||||
#include "json_writer.h"
|
||||
#include "main.h"
|
||||
@ -274,6 +275,327 @@ static void codegen(const char *template, ...)
|
||||
free(s);
|
||||
}
|
||||
|
||||
static void print_hex(const char *data, int data_sz)
|
||||
{
|
||||
int i, len;
|
||||
|
||||
for (i = 0, len = 0; i < data_sz; i++) {
|
||||
int w = data[i] ? 4 : 2;
|
||||
|
||||
len += w;
|
||||
if (len > 78) {
|
||||
printf("\\\n");
|
||||
len = w;
|
||||
}
|
||||
if (!data[i])
|
||||
printf("\\0");
|
||||
else
|
||||
printf("\\x%02x", (unsigned char)data[i]);
|
||||
}
|
||||
}
|
||||
|
||||
static size_t bpf_map_mmap_sz(const struct bpf_map *map)
|
||||
{
|
||||
long page_sz = sysconf(_SC_PAGE_SIZE);
|
||||
size_t map_sz;
|
||||
|
||||
map_sz = (size_t)roundup(bpf_map__value_size(map), 8) * bpf_map__max_entries(map);
|
||||
map_sz = roundup(map_sz, page_sz);
|
||||
return map_sz;
|
||||
}
|
||||
|
||||
static void codegen_attach_detach(struct bpf_object *obj, const char *obj_name)
|
||||
{
|
||||
struct bpf_program *prog;
|
||||
|
||||
bpf_object__for_each_program(prog, obj) {
|
||||
const char *tp_name;
|
||||
|
||||
codegen("\
|
||||
\n\
|
||||
\n\
|
||||
static inline int \n\
|
||||
%1$s__%2$s__attach(struct %1$s *skel) \n\
|
||||
{ \n\
|
||||
int prog_fd = skel->progs.%2$s.prog_fd; \n\
|
||||
", obj_name, bpf_program__name(prog));
|
||||
|
||||
switch (bpf_program__get_type(prog)) {
|
||||
case BPF_PROG_TYPE_RAW_TRACEPOINT:
|
||||
tp_name = strchr(bpf_program__section_name(prog), '/') + 1;
|
||||
printf("\tint fd = bpf_raw_tracepoint_open(\"%s\", prog_fd);\n", tp_name);
|
||||
break;
|
||||
case BPF_PROG_TYPE_TRACING:
|
||||
printf("\tint fd = bpf_raw_tracepoint_open(NULL, prog_fd);\n");
|
||||
break;
|
||||
default:
|
||||
printf("\tint fd = ((void)prog_fd, 0); /* auto-attach not supported */\n");
|
||||
break;
|
||||
}
|
||||
codegen("\
|
||||
\n\
|
||||
\n\
|
||||
if (fd > 0) \n\
|
||||
skel->links.%1$s_fd = fd; \n\
|
||||
return fd; \n\
|
||||
} \n\
|
||||
", bpf_program__name(prog));
|
||||
}
|
||||
|
||||
codegen("\
|
||||
\n\
|
||||
\n\
|
||||
static inline int \n\
|
||||
%1$s__attach(struct %1$s *skel) \n\
|
||||
{ \n\
|
||||
int ret = 0; \n\
|
||||
\n\
|
||||
", obj_name);
|
||||
|
||||
bpf_object__for_each_program(prog, obj) {
|
||||
codegen("\
|
||||
\n\
|
||||
ret = ret < 0 ? ret : %1$s__%2$s__attach(skel); \n\
|
||||
", obj_name, bpf_program__name(prog));
|
||||
}
|
||||
|
||||
codegen("\
|
||||
\n\
|
||||
return ret < 0 ? ret : 0; \n\
|
||||
} \n\
|
||||
\n\
|
||||
static inline void \n\
|
||||
%1$s__detach(struct %1$s *skel) \n\
|
||||
{ \n\
|
||||
", obj_name);
|
||||
|
||||
bpf_object__for_each_program(prog, obj) {
|
||||
codegen("\
|
||||
\n\
|
||||
skel_closenz(skel->links.%1$s_fd); \n\
|
||||
", bpf_program__name(prog));
|
||||
}
|
||||
|
||||
codegen("\
|
||||
\n\
|
||||
} \n\
|
||||
");
|
||||
}
|
||||
|
||||
static void codegen_destroy(struct bpf_object *obj, const char *obj_name)
|
||||
{
|
||||
struct bpf_program *prog;
|
||||
struct bpf_map *map;
|
||||
|
||||
codegen("\
|
||||
\n\
|
||||
static void \n\
|
||||
%1$s__destroy(struct %1$s *skel) \n\
|
||||
{ \n\
|
||||
if (!skel) \n\
|
||||
return; \n\
|
||||
%1$s__detach(skel); \n\
|
||||
",
|
||||
obj_name);
|
||||
|
||||
bpf_object__for_each_program(prog, obj) {
|
||||
codegen("\
|
||||
\n\
|
||||
skel_closenz(skel->progs.%1$s.prog_fd); \n\
|
||||
", bpf_program__name(prog));
|
||||
}
|
||||
|
||||
bpf_object__for_each_map(map, obj) {
|
||||
const char * ident;
|
||||
|
||||
ident = get_map_ident(map);
|
||||
if (!ident)
|
||||
continue;
|
||||
if (bpf_map__is_internal(map) &&
|
||||
(bpf_map__def(map)->map_flags & BPF_F_MMAPABLE))
|
||||
printf("\tmunmap(skel->%1$s, %2$zd);\n",
|
||||
ident, bpf_map_mmap_sz(map));
|
||||
codegen("\
|
||||
\n\
|
||||
skel_closenz(skel->maps.%1$s.map_fd); \n\
|
||||
", ident);
|
||||
}
|
||||
codegen("\
|
||||
\n\
|
||||
free(skel); \n\
|
||||
} \n\
|
||||
",
|
||||
obj_name);
|
||||
}
|
||||
|
||||
static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *header_guard)
|
||||
{
|
||||
struct bpf_object_load_attr load_attr = {};
|
||||
DECLARE_LIBBPF_OPTS(gen_loader_opts, opts);
|
||||
struct bpf_map *map;
|
||||
int err = 0;
|
||||
|
||||
err = bpf_object__gen_loader(obj, &opts);
|
||||
if (err)
|
||||
return err;
|
||||
|
||||
load_attr.obj = obj;
|
||||
if (verifier_logs)
|
||||
/* log_level1 + log_level2 + stats, but not stable UAPI */
|
||||
load_attr.log_level = 1 + 2 + 4;
|
||||
|
||||
err = bpf_object__load_xattr(&load_attr);
|
||||
if (err) {
|
||||
p_err("failed to load object file");
|
||||
goto out;
|
||||
}
|
||||
/* If there was no error during load then gen_loader_opts
|
||||
* are populated with the loader program.
|
||||
*/
|
||||
|
||||
/* finish generating 'struct skel' */
|
||||
codegen("\
|
||||
\n\
|
||||
}; \n\
|
||||
", obj_name);
|
||||
|
||||
|
||||
codegen_attach_detach(obj, obj_name);
|
||||
|
||||
codegen_destroy(obj, obj_name);
|
||||
|
||||
codegen("\
|
||||
\n\
|
||||
static inline struct %1$s * \n\
|
||||
%1$s__open(void) \n\
|
||||
{ \n\
|
||||
struct %1$s *skel; \n\
|
||||
\n\
|
||||
skel = calloc(sizeof(*skel), 1); \n\
|
||||
if (!skel) \n\
|
||||
goto cleanup; \n\
|
||||
skel->ctx.sz = (void *)&skel->links - (void *)skel; \n\
|
||||
",
|
||||
obj_name, opts.data_sz);
|
||||
bpf_object__for_each_map(map, obj) {
|
||||
const char *ident;
|
||||
const void *mmap_data = NULL;
|
||||
size_t mmap_size = 0;
|
||||
|
||||
ident = get_map_ident(map);
|
||||
if (!ident)
|
||||
continue;
|
||||
|
||||
if (!bpf_map__is_internal(map) ||
|
||||
!(bpf_map__def(map)->map_flags & BPF_F_MMAPABLE))
|
||||
continue;
|
||||
|
||||
codegen("\
|
||||
\n\
|
||||
skel->%1$s = \n\
|
||||
mmap(NULL, %2$zd, PROT_READ | PROT_WRITE,\n\
|
||||
MAP_SHARED | MAP_ANONYMOUS, -1, 0); \n\
|
||||
if (skel->%1$s == (void *) -1) \n\
|
||||
goto cleanup; \n\
|
||||
memcpy(skel->%1$s, (void *)\"\\ \n\
|
||||
", ident, bpf_map_mmap_sz(map));
|
||||
mmap_data = bpf_map__initial_value(map, &mmap_size);
|
||||
print_hex(mmap_data, mmap_size);
|
||||
printf("\", %2$zd);\n"
|
||||
"\tskel->maps.%1$s.initial_value = (__u64)(long)skel->%1$s;\n",
|
||||
ident, mmap_size);
|
||||
}
|
||||
codegen("\
|
||||
\n\
|
||||
return skel; \n\
|
||||
cleanup: \n\
|
||||
%1$s__destroy(skel); \n\
|
||||
return NULL; \n\
|
||||
} \n\
|
||||
\n\
|
||||
static inline int \n\
|
||||
%1$s__load(struct %1$s *skel) \n\
|
||||
{ \n\
|
||||
struct bpf_load_and_run_opts opts = {}; \n\
|
||||
int err; \n\
|
||||
\n\
|
||||
opts.ctx = (struct bpf_loader_ctx *)skel; \n\
|
||||
opts.data_sz = %2$d; \n\
|
||||
opts.data = (void *)\"\\ \n\
|
||||
",
|
||||
obj_name, opts.data_sz);
|
||||
print_hex(opts.data, opts.data_sz);
|
||||
codegen("\
|
||||
\n\
|
||||
\"; \n\
|
||||
");
|
||||
|
||||
codegen("\
|
||||
\n\
|
||||
opts.insns_sz = %d; \n\
|
||||
opts.insns = (void *)\"\\ \n\
|
||||
",
|
||||
opts.insns_sz);
|
||||
print_hex(opts.insns, opts.insns_sz);
|
||||
codegen("\
|
||||
\n\
|
||||
\"; \n\
|
||||
err = bpf_load_and_run(&opts); \n\
|
||||
if (err < 0) \n\
|
||||
return err; \n\
|
||||
", obj_name);
|
||||
bpf_object__for_each_map(map, obj) {
|
||||
const char *ident, *mmap_flags;
|
||||
|
||||
ident = get_map_ident(map);
|
||||
if (!ident)
|
||||
continue;
|
||||
|
||||
if (!bpf_map__is_internal(map) ||
|
||||
!(bpf_map__def(map)->map_flags & BPF_F_MMAPABLE))
|
||||
continue;
|
||||
if (bpf_map__def(map)->map_flags & BPF_F_RDONLY_PROG)
|
||||
mmap_flags = "PROT_READ";
|
||||
else
|
||||
mmap_flags = "PROT_READ | PROT_WRITE";
|
||||
|
||||
printf("\tskel->%1$s =\n"
|
||||
"\t\tmmap(skel->%1$s, %2$zd, %3$s, MAP_SHARED | MAP_FIXED,\n"
|
||||
"\t\t\tskel->maps.%1$s.map_fd, 0);\n",
|
||||
ident, bpf_map_mmap_sz(map), mmap_flags);
|
||||
}
|
||||
codegen("\
|
||||
\n\
|
||||
return 0; \n\
|
||||
} \n\
|
||||
\n\
|
||||
static inline struct %1$s * \n\
|
||||
%1$s__open_and_load(void) \n\
|
||||
{ \n\
|
||||
struct %1$s *skel; \n\
|
||||
\n\
|
||||
skel = %1$s__open(); \n\
|
||||
if (!skel) \n\
|
||||
return NULL; \n\
|
||||
if (%1$s__load(skel)) { \n\
|
||||
%1$s__destroy(skel); \n\
|
||||
return NULL; \n\
|
||||
} \n\
|
||||
return skel; \n\
|
||||
} \n\
|
||||
", obj_name);
|
||||
|
||||
codegen("\
|
||||
\n\
|
||||
\n\
|
||||
#endif /* %s */ \n\
|
||||
",
|
||||
header_guard);
|
||||
err = 0;
|
||||
out:
|
||||
return err;
|
||||
}
|
||||
|
||||
static int do_skeleton(int argc, char **argv)
|
||||
{
|
||||
char header_guard[MAX_OBJ_NAME_LEN + sizeof("__SKEL_H__")];
|
||||
@ -283,7 +605,7 @@ static int do_skeleton(int argc, char **argv)
|
||||
struct bpf_object *obj = NULL;
|
||||
const char *file, *ident;
|
||||
struct bpf_program *prog;
|
||||
int fd, len, err = -1;
|
||||
int fd, err = -1;
|
||||
struct bpf_map *map;
|
||||
struct btf *btf;
|
||||
struct stat st;
|
||||
@ -365,7 +687,25 @@ static int do_skeleton(int argc, char **argv)
|
||||
}
|
||||
|
||||
get_header_guard(header_guard, obj_name);
|
||||
codegen("\
|
||||
if (use_loader) {
|
||||
codegen("\
|
||||
\n\
|
||||
/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */ \n\
|
||||
/* THIS FILE IS AUTOGENERATED! */ \n\
|
||||
#ifndef %2$s \n\
|
||||
#define %2$s \n\
|
||||
\n\
|
||||
#include <stdlib.h> \n\
|
||||
#include <bpf/bpf.h> \n\
|
||||
#include <bpf/skel_internal.h> \n\
|
||||
\n\
|
||||
struct %1$s { \n\
|
||||
struct bpf_loader_ctx ctx; \n\
|
||||
",
|
||||
obj_name, header_guard
|
||||
);
|
||||
} else {
|
||||
codegen("\
|
||||
\n\
|
||||
/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */ \n\
|
||||
\n\
|
||||
@ -381,7 +721,8 @@ static int do_skeleton(int argc, char **argv)
|
||||
struct bpf_object *obj; \n\
|
||||
",
|
||||
obj_name, header_guard
|
||||
);
|
||||
);
|
||||
}
|
||||
|
||||
if (map_cnt) {
|
||||
printf("\tstruct {\n");
|
||||
@ -389,7 +730,10 @@ static int do_skeleton(int argc, char **argv)
|
||||
ident = get_map_ident(map);
|
||||
if (!ident)
|
||||
continue;
|
||||
printf("\t\tstruct bpf_map *%s;\n", ident);
|
||||
if (use_loader)
|
||||
printf("\t\tstruct bpf_map_desc %s;\n", ident);
|
||||
else
|
||||
printf("\t\tstruct bpf_map *%s;\n", ident);
|
||||
}
|
||||
printf("\t} maps;\n");
|
||||
}
|
||||
@ -397,14 +741,22 @@ static int do_skeleton(int argc, char **argv)
|
||||
if (prog_cnt) {
|
||||
printf("\tstruct {\n");
|
||||
bpf_object__for_each_program(prog, obj) {
|
||||
printf("\t\tstruct bpf_program *%s;\n",
|
||||
bpf_program__name(prog));
|
||||
if (use_loader)
|
||||
printf("\t\tstruct bpf_prog_desc %s;\n",
|
||||
bpf_program__name(prog));
|
||||
else
|
||||
printf("\t\tstruct bpf_program *%s;\n",
|
||||
bpf_program__name(prog));
|
||||
}
|
||||
printf("\t} progs;\n");
|
||||
printf("\tstruct {\n");
|
||||
bpf_object__for_each_program(prog, obj) {
|
||||
printf("\t\tstruct bpf_link *%s;\n",
|
||||
bpf_program__name(prog));
|
||||
if (use_loader)
|
||||
printf("\t\tint %s_fd;\n",
|
||||
bpf_program__name(prog));
|
||||
else
|
||||
printf("\t\tstruct bpf_link *%s;\n",
|
||||
bpf_program__name(prog));
|
||||
}
|
||||
printf("\t} links;\n");
|
||||
}
|
||||
@ -415,6 +767,10 @@ static int do_skeleton(int argc, char **argv)
|
||||
if (err)
|
||||
goto out;
|
||||
}
|
||||
if (use_loader) {
|
||||
err = gen_trace(obj, obj_name, header_guard);
|
||||
goto out;
|
||||
}
|
||||
|
||||
codegen("\
|
||||
\n\
|
||||
@ -584,19 +940,7 @@ static int do_skeleton(int argc, char **argv)
|
||||
file_sz);
|
||||
|
||||
/* embed contents of BPF object file */
|
||||
for (i = 0, len = 0; i < file_sz; i++) {
|
||||
int w = obj_data[i] ? 4 : 2;
|
||||
|
||||
len += w;
|
||||
if (len > 78) {
|
||||
printf("\\\n");
|
||||
len = w;
|
||||
}
|
||||
if (!obj_data[i])
|
||||
printf("\\0");
|
||||
else
|
||||
printf("\\x%02x", (unsigned char)obj_data[i]);
|
||||
}
|
||||
print_hex(obj_data, file_sz);
|
||||
|
||||
codegen("\
|
||||
\n\
|
||||
|
||||
@ -29,6 +29,7 @@ bool show_pinned;
|
||||
bool block_mount;
|
||||
bool verifier_logs;
|
||||
bool relaxed_maps;
|
||||
bool use_loader;
|
||||
struct btf *base_btf;
|
||||
struct pinned_obj_table prog_table;
|
||||
struct pinned_obj_table map_table;
|
||||
@ -392,6 +393,7 @@ int main(int argc, char **argv)
|
||||
{ "mapcompat", no_argument, NULL, 'm' },
|
||||
{ "nomount", no_argument, NULL, 'n' },
|
||||
{ "debug", no_argument, NULL, 'd' },
|
||||
{ "use-loader", no_argument, NULL, 'L' },
|
||||
{ "base-btf", required_argument, NULL, 'B' },
|
||||
{ 0 }
|
||||
};
|
||||
@ -409,7 +411,7 @@ int main(int argc, char **argv)
|
||||
hash_init(link_table.table);
|
||||
|
||||
opterr = 0;
|
||||
while ((opt = getopt_long(argc, argv, "VhpjfmndB:",
|
||||
while ((opt = getopt_long(argc, argv, "VhpjfLmndB:",
|
||||
options, NULL)) >= 0) {
|
||||
switch (opt) {
|
||||
case 'V':
|
||||
@ -452,6 +454,9 @@ int main(int argc, char **argv)
|
||||
return -1;
|
||||
}
|
||||
break;
|
||||
case 'L':
|
||||
use_loader = true;
|
||||
break;
|
||||
default:
|
||||
p_err("unrecognized option '%s'", argv[optind - 1]);
|
||||
if (json_output)
|
||||
|
||||
@ -90,6 +90,7 @@ extern bool show_pids;
|
||||
extern bool block_mount;
|
||||
extern bool verifier_logs;
|
||||
extern bool relaxed_maps;
|
||||
extern bool use_loader;
|
||||
extern struct btf *base_btf;
|
||||
extern struct pinned_obj_table prog_table;
|
||||
extern struct pinned_obj_table map_table;
|
||||
|
||||
@ -16,6 +16,7 @@
|
||||
#include <sys/types.h>
|
||||
#include <sys/stat.h>
|
||||
#include <sys/syscall.h>
|
||||
#include <dirent.h>
|
||||
|
||||
#include <linux/err.h>
|
||||
#include <linux/perf_event.h>
|
||||
@ -24,6 +25,8 @@
|
||||
#include <bpf/bpf.h>
|
||||
#include <bpf/btf.h>
|
||||
#include <bpf/libbpf.h>
|
||||
#include <bpf/bpf_gen_internal.h>
|
||||
#include <bpf/skel_internal.h>
|
||||
|
||||
#include "cfg.h"
|
||||
#include "main.h"
|
||||
@ -1499,7 +1502,7 @@ static int load_with_options(int argc, char **argv, bool first_prog_only)
|
||||
set_max_rlimit();
|
||||
|
||||
obj = bpf_object__open_file(file, &open_opts);
|
||||
if (IS_ERR_OR_NULL(obj)) {
|
||||
if (libbpf_get_error(obj)) {
|
||||
p_err("failed to open object file");
|
||||
goto err_free_reuse_maps;
|
||||
}
|
||||
@ -1645,8 +1648,110 @@ err_free_reuse_maps:
|
||||
return -1;
|
||||
}
|
||||
|
||||
static int count_open_fds(void)
|
||||
{
|
||||
DIR *dp = opendir("/proc/self/fd");
|
||||
struct dirent *de;
|
||||
int cnt = -3;
|
||||
|
||||
if (!dp)
|
||||
return -1;
|
||||
|
||||
while ((de = readdir(dp)))
|
||||
cnt++;
|
||||
|
||||
closedir(dp);
|
||||
return cnt;
|
||||
}
|
||||
|
||||
static int try_loader(struct gen_loader_opts *gen)
|
||||
{
|
||||
struct bpf_load_and_run_opts opts = {};
|
||||
struct bpf_loader_ctx *ctx;
|
||||
int ctx_sz = sizeof(*ctx) + 64 * max(sizeof(struct bpf_map_desc),
|
||||
sizeof(struct bpf_prog_desc));
|
||||
int log_buf_sz = (1u << 24) - 1;
|
||||
int err, fds_before, fd_delta;
|
||||
char *log_buf;
|
||||
|
||||
ctx = alloca(ctx_sz);
|
||||
memset(ctx, 0, ctx_sz);
|
||||
ctx->sz = ctx_sz;
|
||||
ctx->log_level = 1;
|
||||
ctx->log_size = log_buf_sz;
|
||||
log_buf = malloc(log_buf_sz);
|
||||
if (!log_buf)
|
||||
return -ENOMEM;
|
||||
ctx->log_buf = (long) log_buf;
|
||||
opts.ctx = ctx;
|
||||
opts.data = gen->data;
|
||||
opts.data_sz = gen->data_sz;
|
||||
opts.insns = gen->insns;
|
||||
opts.insns_sz = gen->insns_sz;
|
||||
fds_before = count_open_fds();
|
||||
err = bpf_load_and_run(&opts);
|
||||
fd_delta = count_open_fds() - fds_before;
|
||||
if (err < 0) {
|
||||
fprintf(stderr, "err %d\n%s\n%s", err, opts.errstr, log_buf);
|
||||
if (fd_delta)
|
||||
fprintf(stderr, "loader prog leaked %d FDs\n",
|
||||
fd_delta);
|
||||
}
|
||||
free(log_buf);
|
||||
return err;
|
||||
}
|
||||
|
||||
static int do_loader(int argc, char **argv)
|
||||
{
|
||||
DECLARE_LIBBPF_OPTS(bpf_object_open_opts, open_opts);
|
||||
DECLARE_LIBBPF_OPTS(gen_loader_opts, gen);
|
||||
struct bpf_object_load_attr load_attr = {};
|
||||
struct bpf_object *obj;
|
||||
const char *file;
|
||||
int err = 0;
|
||||
|
||||
if (!REQ_ARGS(1))
|
||||
return -1;
|
||||
file = GET_ARG();
|
||||
|
||||
obj = bpf_object__open_file(file, &open_opts);
|
||||
if (libbpf_get_error(obj)) {
|
||||
p_err("failed to open object file");
|
||||
goto err_close_obj;
|
||||
}
|
||||
|
||||
err = bpf_object__gen_loader(obj, &gen);
|
||||
if (err)
|
||||
goto err_close_obj;
|
||||
|
||||
load_attr.obj = obj;
|
||||
if (verifier_logs)
|
||||
/* log_level1 + log_level2 + stats, but not stable UAPI */
|
||||
load_attr.log_level = 1 + 2 + 4;
|
||||
|
||||
err = bpf_object__load_xattr(&load_attr);
|
||||
if (err) {
|
||||
p_err("failed to load object file");
|
||||
goto err_close_obj;
|
||||
}
|
||||
|
||||
if (verifier_logs) {
|
||||
struct dump_data dd = {};
|
||||
|
||||
kernel_syms_load(&dd);
|
||||
dump_xlated_plain(&dd, (void *)gen.insns, gen.insns_sz, false, false);
|
||||
kernel_syms_destroy(&dd);
|
||||
}
|
||||
err = try_loader(&gen);
|
||||
err_close_obj:
|
||||
bpf_object__close(obj);
|
||||
return err;
|
||||
}
|
||||
|
||||
static int do_load(int argc, char **argv)
|
||||
{
|
||||
if (use_loader)
|
||||
return do_loader(argc, argv);
|
||||
return load_with_options(argc, argv, true);
|
||||
}
|
||||
|
||||
|
||||
@ -196,6 +196,9 @@ static const char *print_imm(void *private_data,
|
||||
else if (insn->src_reg == BPF_PSEUDO_MAP_VALUE)
|
||||
snprintf(dd->scratch_buff, sizeof(dd->scratch_buff),
|
||||
"map[id:%u][0]+%u", insn->imm, (insn + 1)->imm);
|
||||
else if (insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE)
|
||||
snprintf(dd->scratch_buff, sizeof(dd->scratch_buff),
|
||||
"map[idx:%u]+%u", insn->imm, (insn + 1)->imm);
|
||||
else if (insn->src_reg == BPF_PSEUDO_FUNC)
|
||||
snprintf(dd->scratch_buff, sizeof(dd->scratch_buff),
|
||||
"subprog[%+d]", insn->imm);
|
||||
|
||||
@ -937,6 +937,7 @@ enum bpf_prog_type {
|
||||
BPF_PROG_TYPE_EXT,
|
||||
BPF_PROG_TYPE_LSM,
|
||||
BPF_PROG_TYPE_SK_LOOKUP,
|
||||
BPF_PROG_TYPE_SYSCALL, /* a program that can execute syscalls */
|
||||
};
|
||||
|
||||
enum bpf_attach_type {
|
||||
@ -1097,8 +1098,8 @@ enum bpf_link_type {
|
||||
/* When BPF ldimm64's insn[0].src_reg != 0 then this can have
|
||||
* the following extensions:
|
||||
*
|
||||
* insn[0].src_reg: BPF_PSEUDO_MAP_FD
|
||||
* insn[0].imm: map fd
|
||||
* insn[0].src_reg: BPF_PSEUDO_MAP_[FD|IDX]
|
||||
* insn[0].imm: map fd or fd_idx
|
||||
* insn[1].imm: 0
|
||||
* insn[0].off: 0
|
||||
* insn[1].off: 0
|
||||
@ -1106,15 +1107,19 @@ enum bpf_link_type {
|
||||
* verifier type: CONST_PTR_TO_MAP
|
||||
*/
|
||||
#define BPF_PSEUDO_MAP_FD 1
|
||||
/* insn[0].src_reg: BPF_PSEUDO_MAP_VALUE
|
||||
* insn[0].imm: map fd
|
||||
#define BPF_PSEUDO_MAP_IDX 5
|
||||
|
||||
/* insn[0].src_reg: BPF_PSEUDO_MAP_[IDX_]VALUE
|
||||
* insn[0].imm: map fd or fd_idx
|
||||
* insn[1].imm: offset into value
|
||||
* insn[0].off: 0
|
||||
* insn[1].off: 0
|
||||
* ldimm64 rewrite: address of map[0]+offset
|
||||
* verifier type: PTR_TO_MAP_VALUE
|
||||
*/
|
||||
#define BPF_PSEUDO_MAP_VALUE 2
|
||||
#define BPF_PSEUDO_MAP_VALUE 2
|
||||
#define BPF_PSEUDO_MAP_IDX_VALUE 6
|
||||
|
||||
/* insn[0].src_reg: BPF_PSEUDO_BTF_ID
|
||||
* insn[0].imm: kernel btd id of VAR
|
||||
* insn[1].imm: 0
|
||||
@ -1314,6 +1319,8 @@ union bpf_attr {
|
||||
/* or valid module BTF object fd or 0 to attach to vmlinux */
|
||||
__u32 attach_btf_obj_fd;
|
||||
};
|
||||
__u32 :32; /* pad */
|
||||
__aligned_u64 fd_array; /* array of FDs */
|
||||
};
|
||||
|
||||
struct { /* anonymous struct used by BPF_OBJ_* commands */
|
||||
@ -4735,6 +4742,24 @@ union bpf_attr {
|
||||
* be zero-terminated except when **str_size** is 0.
|
||||
*
|
||||
* Or **-EBUSY** if the per-CPU memory copy buffer is busy.
|
||||
*
|
||||
* long bpf_sys_bpf(u32 cmd, void *attr, u32 attr_size)
|
||||
* Description
|
||||
* Execute bpf syscall with given arguments.
|
||||
* Return
|
||||
* A syscall result.
|
||||
*
|
||||
* long bpf_btf_find_by_name_kind(char *name, int name_sz, u32 kind, int flags)
|
||||
* Description
|
||||
* Find BTF type with given name and kind in vmlinux BTF or in module's BTFs.
|
||||
* Return
|
||||
* Returns btf_id and btf_obj_fd in lower and upper 32 bits.
|
||||
*
|
||||
* long bpf_sys_close(u32 fd)
|
||||
* Description
|
||||
* Execute close syscall for given FD.
|
||||
* Return
|
||||
* A syscall result.
|
||||
*/
|
||||
#define __BPF_FUNC_MAPPER(FN) \
|
||||
FN(unspec), \
|
||||
@ -4903,6 +4928,9 @@ union bpf_attr {
|
||||
FN(check_mtu), \
|
||||
FN(for_each_map_elem), \
|
||||
FN(snprintf), \
|
||||
FN(sys_bpf), \
|
||||
FN(btf_find_by_name_kind), \
|
||||
FN(sys_close), \
|
||||
/* */
|
||||
|
||||
/* integer value in 'imm' field of BPF_CALL instruction selects which helper
|
||||
|
||||
@ -1,3 +1,3 @@
|
||||
libbpf-y := libbpf.o bpf.o nlattr.o btf.o libbpf_errno.o str_error.o \
|
||||
netlink.o bpf_prog_linfo.o libbpf_probes.o xsk.o hashmap.o \
|
||||
btf_dump.o ringbuf.o strset.o linker.o
|
||||
btf_dump.o ringbuf.o strset.o linker.o gen_loader.o
|
||||
|
||||
41
tools/lib/bpf/bpf_gen_internal.h
Normal file
41
tools/lib/bpf/bpf_gen_internal.h
Normal file
@ -0,0 +1,41 @@
|
||||
/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */
|
||||
/* Copyright (c) 2021 Facebook */
|
||||
#ifndef __BPF_GEN_INTERNAL_H
|
||||
#define __BPF_GEN_INTERNAL_H
|
||||
|
||||
struct ksym_relo_desc {
|
||||
const char *name;
|
||||
int kind;
|
||||
int insn_idx;
|
||||
};
|
||||
|
||||
struct bpf_gen {
|
||||
struct gen_loader_opts *opts;
|
||||
void *data_start;
|
||||
void *data_cur;
|
||||
void *insn_start;
|
||||
void *insn_cur;
|
||||
ssize_t cleanup_label;
|
||||
__u32 nr_progs;
|
||||
__u32 nr_maps;
|
||||
int log_level;
|
||||
int error;
|
||||
struct ksym_relo_desc *relos;
|
||||
int relo_cnt;
|
||||
char attach_target[128];
|
||||
int attach_kind;
|
||||
};
|
||||
|
||||
void bpf_gen__init(struct bpf_gen *gen, int log_level);
|
||||
int bpf_gen__finish(struct bpf_gen *gen);
|
||||
void bpf_gen__free(struct bpf_gen *gen);
|
||||
void bpf_gen__load_btf(struct bpf_gen *gen, const void *raw_data, __u32 raw_size);
|
||||
void bpf_gen__map_create(struct bpf_gen *gen, struct bpf_create_map_attr *map_attr, int map_idx);
|
||||
struct bpf_prog_load_params;
|
||||
void bpf_gen__prog_load(struct bpf_gen *gen, struct bpf_prog_load_params *load_attr, int prog_idx);
|
||||
void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *value, __u32 value_size);
|
||||
void bpf_gen__map_freeze(struct bpf_gen *gen, int map_idx);
|
||||
void bpf_gen__record_attach_target(struct bpf_gen *gen, const char *name, enum bpf_attach_type type);
|
||||
void bpf_gen__record_extern(struct bpf_gen *gen, const char *name, int kind, int insn_idx);
|
||||
|
||||
#endif
|
||||
729
tools/lib/bpf/gen_loader.c
Normal file
729
tools/lib/bpf/gen_loader.c
Normal file
@ -0,0 +1,729 @@
|
||||
// SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
|
||||
/* Copyright (c) 2021 Facebook */
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <errno.h>
|
||||
#include <linux/filter.h>
|
||||
#include "btf.h"
|
||||
#include "bpf.h"
|
||||
#include "libbpf.h"
|
||||
#include "libbpf_internal.h"
|
||||
#include "hashmap.h"
|
||||
#include "bpf_gen_internal.h"
|
||||
#include "skel_internal.h"
|
||||
|
||||
#define MAX_USED_MAPS 64
|
||||
#define MAX_USED_PROGS 32
|
||||
|
||||
/* The following structure describes the stack layout of the loader program.
|
||||
* In addition R6 contains the pointer to context.
|
||||
* R7 contains the result of the last sys_bpf command (typically error or FD).
|
||||
* R9 contains the result of the last sys_close command.
|
||||
*
|
||||
* Naming convention:
|
||||
* ctx - bpf program context
|
||||
* stack - bpf program stack
|
||||
* blob - bpf_attr-s, strings, insns, map data.
|
||||
* All the bytes that loader prog will use for read/write.
|
||||
*/
|
||||
struct loader_stack {
|
||||
__u32 btf_fd;
|
||||
__u32 map_fd[MAX_USED_MAPS];
|
||||
__u32 prog_fd[MAX_USED_PROGS];
|
||||
__u32 inner_map_fd;
|
||||
};
|
||||
|
||||
#define stack_off(field) \
|
||||
(__s16)(-sizeof(struct loader_stack) + offsetof(struct loader_stack, field))
|
||||
|
||||
#define attr_field(attr, field) (attr + offsetof(union bpf_attr, field))
|
||||
|
||||
static int realloc_insn_buf(struct bpf_gen *gen, __u32 size)
|
||||
{
|
||||
size_t off = gen->insn_cur - gen->insn_start;
|
||||
void *insn_start;
|
||||
|
||||
if (gen->error)
|
||||
return gen->error;
|
||||
if (size > INT32_MAX || off + size > INT32_MAX) {
|
||||
gen->error = -ERANGE;
|
||||
return -ERANGE;
|
||||
}
|
||||
insn_start = realloc(gen->insn_start, off + size);
|
||||
if (!insn_start) {
|
||||
gen->error = -ENOMEM;
|
||||
free(gen->insn_start);
|
||||
gen->insn_start = NULL;
|
||||
return -ENOMEM;
|
||||
}
|
||||
gen->insn_start = insn_start;
|
||||
gen->insn_cur = insn_start + off;
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int realloc_data_buf(struct bpf_gen *gen, __u32 size)
|
||||
{
|
||||
size_t off = gen->data_cur - gen->data_start;
|
||||
void *data_start;
|
||||
|
||||
if (gen->error)
|
||||
return gen->error;
|
||||
if (size > INT32_MAX || off + size > INT32_MAX) {
|
||||
gen->error = -ERANGE;
|
||||
return -ERANGE;
|
||||
}
|
||||
data_start = realloc(gen->data_start, off + size);
|
||||
if (!data_start) {
|
||||
gen->error = -ENOMEM;
|
||||
free(gen->data_start);
|
||||
gen->data_start = NULL;
|
||||
return -ENOMEM;
|
||||
}
|
||||
gen->data_start = data_start;
|
||||
gen->data_cur = data_start + off;
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void emit(struct bpf_gen *gen, struct bpf_insn insn)
|
||||
{
|
||||
if (realloc_insn_buf(gen, sizeof(insn)))
|
||||
return;
|
||||
memcpy(gen->insn_cur, &insn, sizeof(insn));
|
||||
gen->insn_cur += sizeof(insn);
|
||||
}
|
||||
|
||||
static void emit2(struct bpf_gen *gen, struct bpf_insn insn1, struct bpf_insn insn2)
|
||||
{
|
||||
emit(gen, insn1);
|
||||
emit(gen, insn2);
|
||||
}
|
||||
|
||||
void bpf_gen__init(struct bpf_gen *gen, int log_level)
|
||||
{
|
||||
size_t stack_sz = sizeof(struct loader_stack);
|
||||
int i;
|
||||
|
||||
gen->log_level = log_level;
|
||||
/* save ctx pointer into R6 */
|
||||
emit(gen, BPF_MOV64_REG(BPF_REG_6, BPF_REG_1));
|
||||
|
||||
/* bzero stack */
|
||||
emit(gen, BPF_MOV64_REG(BPF_REG_1, BPF_REG_10));
|
||||
emit(gen, BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, -stack_sz));
|
||||
emit(gen, BPF_MOV64_IMM(BPF_REG_2, stack_sz));
|
||||
emit(gen, BPF_MOV64_IMM(BPF_REG_3, 0));
|
||||
emit(gen, BPF_EMIT_CALL(BPF_FUNC_probe_read_kernel));
|
||||
|
||||
/* jump over cleanup code */
|
||||
emit(gen, BPF_JMP_IMM(BPF_JA, 0, 0,
|
||||
/* size of cleanup code below */
|
||||
(stack_sz / 4) * 3 + 2));
|
||||
|
||||
/* remember the label where all error branches will jump to */
|
||||
gen->cleanup_label = gen->insn_cur - gen->insn_start;
|
||||
/* emit cleanup code: close all temp FDs */
|
||||
for (i = 0; i < stack_sz; i += 4) {
|
||||
emit(gen, BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_10, -stack_sz + i));
|
||||
emit(gen, BPF_JMP_IMM(BPF_JSLE, BPF_REG_1, 0, 1));
|
||||
emit(gen, BPF_EMIT_CALL(BPF_FUNC_sys_close));
|
||||
}
|
||||
/* R7 contains the error code from sys_bpf. Copy it into R0 and exit. */
|
||||
emit(gen, BPF_MOV64_REG(BPF_REG_0, BPF_REG_7));
|
||||
emit(gen, BPF_EXIT_INSN());
|
||||
}
|
||||
|
||||
static int add_data(struct bpf_gen *gen, const void *data, __u32 size)
|
||||
{
|
||||
void *prev;
|
||||
|
||||
if (realloc_data_buf(gen, size))
|
||||
return 0;
|
||||
prev = gen->data_cur;
|
||||
memcpy(gen->data_cur, data, size);
|
||||
gen->data_cur += size;
|
||||
return prev - gen->data_start;
|
||||
}
|
||||
|
||||
static int insn_bytes_to_bpf_size(__u32 sz)
|
||||
{
|
||||
switch (sz) {
|
||||
case 8: return BPF_DW;
|
||||
case 4: return BPF_W;
|
||||
case 2: return BPF_H;
|
||||
case 1: return BPF_B;
|
||||
default: return -1;
|
||||
}
|
||||
}
|
||||
|
||||
/* *(u64 *)(blob + off) = (u64)(void *)(blob + data) */
|
||||
static void emit_rel_store(struct bpf_gen *gen, int off, int data)
|
||||
{
|
||||
emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_0, BPF_PSEUDO_MAP_IDX_VALUE,
|
||||
0, 0, 0, data));
|
||||
emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE,
|
||||
0, 0, 0, off));
|
||||
emit(gen, BPF_STX_MEM(BPF_DW, BPF_REG_1, BPF_REG_0, 0));
|
||||
}
|
||||
|
||||
/* *(u64 *)(blob + off) = (u64)(void *)(%sp + stack_off) */
|
||||
static void emit_rel_store_sp(struct bpf_gen *gen, int off, int stack_off)
|
||||
{
|
||||
emit(gen, BPF_MOV64_REG(BPF_REG_0, BPF_REG_10));
|
||||
emit(gen, BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, stack_off));
|
||||
emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE,
|
||||
0, 0, 0, off));
|
||||
emit(gen, BPF_STX_MEM(BPF_DW, BPF_REG_1, BPF_REG_0, 0));
|
||||
}
|
||||
|
||||
static void move_ctx2blob(struct bpf_gen *gen, int off, int size, int ctx_off,
|
||||
bool check_non_zero)
|
||||
{
|
||||
emit(gen, BPF_LDX_MEM(insn_bytes_to_bpf_size(size), BPF_REG_0, BPF_REG_6, ctx_off));
|
||||
if (check_non_zero)
|
||||
/* If value in ctx is zero don't update the blob.
|
||||
* For example: when ctx->map.max_entries == 0, keep default max_entries from bpf.c
|
||||
*/
|
||||
emit(gen, BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 3));
|
||||
emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE,
|
||||
0, 0, 0, off));
|
||||
emit(gen, BPF_STX_MEM(insn_bytes_to_bpf_size(size), BPF_REG_1, BPF_REG_0, 0));
|
||||
}
|
||||
|
||||
static void move_stack2blob(struct bpf_gen *gen, int off, int size, int stack_off)
|
||||
{
|
||||
emit(gen, BPF_LDX_MEM(insn_bytes_to_bpf_size(size), BPF_REG_0, BPF_REG_10, stack_off));
|
||||
emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE,
|
||||
0, 0, 0, off));
|
||||
emit(gen, BPF_STX_MEM(insn_bytes_to_bpf_size(size), BPF_REG_1, BPF_REG_0, 0));
|
||||
}
|
||||
|
||||
static void move_stack2ctx(struct bpf_gen *gen, int ctx_off, int size, int stack_off)
|
||||
{
|
||||
emit(gen, BPF_LDX_MEM(insn_bytes_to_bpf_size(size), BPF_REG_0, BPF_REG_10, stack_off));
|
||||
emit(gen, BPF_STX_MEM(insn_bytes_to_bpf_size(size), BPF_REG_6, BPF_REG_0, ctx_off));
|
||||
}
|
||||
|
||||
static void emit_sys_bpf(struct bpf_gen *gen, int cmd, int attr, int attr_size)
|
||||
{
|
||||
emit(gen, BPF_MOV64_IMM(BPF_REG_1, cmd));
|
||||
emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_2, BPF_PSEUDO_MAP_IDX_VALUE,
|
||||
0, 0, 0, attr));
|
||||
emit(gen, BPF_MOV64_IMM(BPF_REG_3, attr_size));
|
||||
emit(gen, BPF_EMIT_CALL(BPF_FUNC_sys_bpf));
|
||||
/* remember the result in R7 */
|
||||
emit(gen, BPF_MOV64_REG(BPF_REG_7, BPF_REG_0));
|
||||
}
|
||||
|
||||
static bool is_simm16(__s64 value)
|
||||
{
|
||||
return value == (__s64)(__s16)value;
|
||||
}
|
||||
|
||||
static void emit_check_err(struct bpf_gen *gen)
|
||||
{
|
||||
__s64 off = -(gen->insn_cur - gen->insn_start - gen->cleanup_label) / 8 - 1;
|
||||
|
||||
/* R7 contains result of last sys_bpf command.
|
||||
* if (R7 < 0) goto cleanup;
|
||||
*/
|
||||
if (is_simm16(off)) {
|
||||
emit(gen, BPF_JMP_IMM(BPF_JSLT, BPF_REG_7, 0, off));
|
||||
} else {
|
||||
gen->error = -ERANGE;
|
||||
emit(gen, BPF_JMP_IMM(BPF_JA, 0, 0, -1));
|
||||
}
|
||||
}
|
||||
|
||||
/* reg1 and reg2 should not be R1 - R5. They can be R0, R6 - R10 */
|
||||
static void emit_debug(struct bpf_gen *gen, int reg1, int reg2,
|
||||
const char *fmt, va_list args)
|
||||
{
|
||||
char buf[1024];
|
||||
int addr, len, ret;
|
||||
|
||||
if (!gen->log_level)
|
||||
return;
|
||||
ret = vsnprintf(buf, sizeof(buf), fmt, args);
|
||||
if (ret < 1024 - 7 && reg1 >= 0 && reg2 < 0)
|
||||
/* The special case to accommodate common debug_ret():
|
||||
* to avoid specifying BPF_REG_7 and adding " r=%%d" to
|
||||
* prints explicitly.
|
||||
*/
|
||||
strcat(buf, " r=%d");
|
||||
len = strlen(buf) + 1;
|
||||
addr = add_data(gen, buf, len);
|
||||
|
||||
emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE,
|
||||
0, 0, 0, addr));
|
||||
emit(gen, BPF_MOV64_IMM(BPF_REG_2, len));
|
||||
if (reg1 >= 0)
|
||||
emit(gen, BPF_MOV64_REG(BPF_REG_3, reg1));
|
||||
if (reg2 >= 0)
|
||||
emit(gen, BPF_MOV64_REG(BPF_REG_4, reg2));
|
||||
emit(gen, BPF_EMIT_CALL(BPF_FUNC_trace_printk));
|
||||
}
|
||||
|
||||
static void debug_regs(struct bpf_gen *gen, int reg1, int reg2, const char *fmt, ...)
|
||||
{
|
||||
va_list args;
|
||||
|
||||
va_start(args, fmt);
|
||||
emit_debug(gen, reg1, reg2, fmt, args);
|
||||
va_end(args);
|
||||
}
|
||||
|
||||
static void debug_ret(struct bpf_gen *gen, const char *fmt, ...)
|
||||
{
|
||||
va_list args;
|
||||
|
||||
va_start(args, fmt);
|
||||
emit_debug(gen, BPF_REG_7, -1, fmt, args);
|
||||
va_end(args);
|
||||
}
|
||||
|
||||
static void __emit_sys_close(struct bpf_gen *gen)
|
||||
{
|
||||
emit(gen, BPF_JMP_IMM(BPF_JSLE, BPF_REG_1, 0,
|
||||
/* 2 is the number of the following insns
|
||||
* * 6 is additional insns in debug_regs
|
||||
*/
|
||||
2 + (gen->log_level ? 6 : 0)));
|
||||
emit(gen, BPF_MOV64_REG(BPF_REG_9, BPF_REG_1));
|
||||
emit(gen, BPF_EMIT_CALL(BPF_FUNC_sys_close));
|
||||
debug_regs(gen, BPF_REG_9, BPF_REG_0, "close(%%d) = %%d");
|
||||
}
|
||||
|
||||
static void emit_sys_close_stack(struct bpf_gen *gen, int stack_off)
|
||||
{
|
||||
emit(gen, BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_10, stack_off));
|
||||
__emit_sys_close(gen);
|
||||
}
|
||||
|
||||
static void emit_sys_close_blob(struct bpf_gen *gen, int blob_off)
|
||||
{
|
||||
emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_0, BPF_PSEUDO_MAP_IDX_VALUE,
|
||||
0, 0, 0, blob_off));
|
||||
emit(gen, BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 0));
|
||||
__emit_sys_close(gen);
|
||||
}
|
||||
|
||||
int bpf_gen__finish(struct bpf_gen *gen)
|
||||
{
|
||||
int i;
|
||||
|
||||
emit_sys_close_stack(gen, stack_off(btf_fd));
|
||||
for (i = 0; i < gen->nr_progs; i++)
|
||||
move_stack2ctx(gen,
|
||||
sizeof(struct bpf_loader_ctx) +
|
||||
sizeof(struct bpf_map_desc) * gen->nr_maps +
|
||||
sizeof(struct bpf_prog_desc) * i +
|
||||
offsetof(struct bpf_prog_desc, prog_fd), 4,
|
||||
stack_off(prog_fd[i]));
|
||||
for (i = 0; i < gen->nr_maps; i++)
|
||||
move_stack2ctx(gen,
|
||||
sizeof(struct bpf_loader_ctx) +
|
||||
sizeof(struct bpf_map_desc) * i +
|
||||
offsetof(struct bpf_map_desc, map_fd), 4,
|
||||
stack_off(map_fd[i]));
|
||||
emit(gen, BPF_MOV64_IMM(BPF_REG_0, 0));
|
||||
emit(gen, BPF_EXIT_INSN());
|
||||
pr_debug("gen: finish %d\n", gen->error);
|
||||
if (!gen->error) {
|
||||
struct gen_loader_opts *opts = gen->opts;
|
||||
|
||||
opts->insns = gen->insn_start;
|
||||
opts->insns_sz = gen->insn_cur - gen->insn_start;
|
||||
opts->data = gen->data_start;
|
||||
opts->data_sz = gen->data_cur - gen->data_start;
|
||||
}
|
||||
return gen->error;
|
||||
}
|
||||
|
||||
void bpf_gen__free(struct bpf_gen *gen)
|
||||
{
|
||||
if (!gen)
|
||||
return;
|
||||
free(gen->data_start);
|
||||
free(gen->insn_start);
|
||||
free(gen);
|
||||
}
|
||||
|
||||
void bpf_gen__load_btf(struct bpf_gen *gen, const void *btf_raw_data,
|
||||
__u32 btf_raw_size)
|
||||
{
|
||||
int attr_size = offsetofend(union bpf_attr, btf_log_level);
|
||||
int btf_data, btf_load_attr;
|
||||
union bpf_attr attr;
|
||||
|
||||
memset(&attr, 0, attr_size);
|
||||
pr_debug("gen: load_btf: size %d\n", btf_raw_size);
|
||||
btf_data = add_data(gen, btf_raw_data, btf_raw_size);
|
||||
|
||||
attr.btf_size = btf_raw_size;
|
||||
btf_load_attr = add_data(gen, &attr, attr_size);
|
||||
|
||||
/* populate union bpf_attr with user provided log details */
|
||||
move_ctx2blob(gen, attr_field(btf_load_attr, btf_log_level), 4,
|
||||
offsetof(struct bpf_loader_ctx, log_level), false);
|
||||
move_ctx2blob(gen, attr_field(btf_load_attr, btf_log_size), 4,
|
||||
offsetof(struct bpf_loader_ctx, log_size), false);
|
||||
move_ctx2blob(gen, attr_field(btf_load_attr, btf_log_buf), 8,
|
||||
offsetof(struct bpf_loader_ctx, log_buf), false);
|
||||
/* populate union bpf_attr with a pointer to the BTF data */
|
||||
emit_rel_store(gen, attr_field(btf_load_attr, btf), btf_data);
|
||||
/* emit BTF_LOAD command */
|
||||
emit_sys_bpf(gen, BPF_BTF_LOAD, btf_load_attr, attr_size);
|
||||
debug_ret(gen, "btf_load size %d", btf_raw_size);
|
||||
emit_check_err(gen);
|
||||
/* remember btf_fd in the stack, if successful */
|
||||
emit(gen, BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_7, stack_off(btf_fd)));
|
||||
}
|
||||
|
||||
void bpf_gen__map_create(struct bpf_gen *gen,
|
||||
struct bpf_create_map_attr *map_attr, int map_idx)
|
||||
{
|
||||
int attr_size = offsetofend(union bpf_attr, btf_vmlinux_value_type_id);
|
||||
bool close_inner_map_fd = false;
|
||||
int map_create_attr;
|
||||
union bpf_attr attr;
|
||||
|
||||
memset(&attr, 0, attr_size);
|
||||
attr.map_type = map_attr->map_type;
|
||||
attr.key_size = map_attr->key_size;
|
||||
attr.value_size = map_attr->value_size;
|
||||
attr.map_flags = map_attr->map_flags;
|
||||
memcpy(attr.map_name, map_attr->name,
|
||||
min((unsigned)strlen(map_attr->name), BPF_OBJ_NAME_LEN - 1));
|
||||
attr.numa_node = map_attr->numa_node;
|
||||
attr.map_ifindex = map_attr->map_ifindex;
|
||||
attr.max_entries = map_attr->max_entries;
|
||||
switch (attr.map_type) {
|
||||
case BPF_MAP_TYPE_PERF_EVENT_ARRAY:
|
||||
case BPF_MAP_TYPE_CGROUP_ARRAY:
|
||||
case BPF_MAP_TYPE_STACK_TRACE:
|
||||
case BPF_MAP_TYPE_ARRAY_OF_MAPS:
|
||||
case BPF_MAP_TYPE_HASH_OF_MAPS:
|
||||
case BPF_MAP_TYPE_DEVMAP:
|
||||
case BPF_MAP_TYPE_DEVMAP_HASH:
|
||||
case BPF_MAP_TYPE_CPUMAP:
|
||||
case BPF_MAP_TYPE_XSKMAP:
|
||||
case BPF_MAP_TYPE_SOCKMAP:
|
||||
case BPF_MAP_TYPE_SOCKHASH:
|
||||
case BPF_MAP_TYPE_QUEUE:
|
||||
case BPF_MAP_TYPE_STACK:
|
||||
case BPF_MAP_TYPE_RINGBUF:
|
||||
break;
|
||||
default:
|
||||
attr.btf_key_type_id = map_attr->btf_key_type_id;
|
||||
attr.btf_value_type_id = map_attr->btf_value_type_id;
|
||||
}
|
||||
|
||||
pr_debug("gen: map_create: %s idx %d type %d value_type_id %d\n",
|
||||
attr.map_name, map_idx, map_attr->map_type, attr.btf_value_type_id);
|
||||
|
||||
map_create_attr = add_data(gen, &attr, attr_size);
|
||||
if (attr.btf_value_type_id)
|
||||
/* populate union bpf_attr with btf_fd saved in the stack earlier */
|
||||
move_stack2blob(gen, attr_field(map_create_attr, btf_fd), 4,
|
||||
stack_off(btf_fd));
|
||||
switch (attr.map_type) {
|
||||
case BPF_MAP_TYPE_ARRAY_OF_MAPS:
|
||||
case BPF_MAP_TYPE_HASH_OF_MAPS:
|
||||
move_stack2blob(gen, attr_field(map_create_attr, inner_map_fd), 4,
|
||||
stack_off(inner_map_fd));
|
||||
close_inner_map_fd = true;
|
||||
break;
|
||||
default:
|
||||
break;
|
||||
}
|
||||
/* conditionally update max_entries */
|
||||
if (map_idx >= 0)
|
||||
move_ctx2blob(gen, attr_field(map_create_attr, max_entries), 4,
|
||||
sizeof(struct bpf_loader_ctx) +
|
||||
sizeof(struct bpf_map_desc) * map_idx +
|
||||
offsetof(struct bpf_map_desc, max_entries),
|
||||
true /* check that max_entries != 0 */);
|
||||
/* emit MAP_CREATE command */
|
||||
emit_sys_bpf(gen, BPF_MAP_CREATE, map_create_attr, attr_size);
|
||||
debug_ret(gen, "map_create %s idx %d type %d value_size %d value_btf_id %d",
|
||||
attr.map_name, map_idx, map_attr->map_type, attr.value_size,
|
||||
attr.btf_value_type_id);
|
||||
emit_check_err(gen);
|
||||
/* remember map_fd in the stack, if successful */
|
||||
if (map_idx < 0) {
|
||||
/* This bpf_gen__map_create() function is called with map_idx >= 0
|
||||
* for all maps that libbpf loading logic tracks.
|
||||
* It's called with -1 to create an inner map.
|
||||
*/
|
||||
emit(gen, BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_7,
|
||||
stack_off(inner_map_fd)));
|
||||
} else if (map_idx != gen->nr_maps) {
|
||||
gen->error = -EDOM; /* internal bug */
|
||||
return;
|
||||
} else {
|
||||
emit(gen, BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_7,
|
||||
stack_off(map_fd[map_idx])));
|
||||
gen->nr_maps++;
|
||||
}
|
||||
if (close_inner_map_fd)
|
||||
emit_sys_close_stack(gen, stack_off(inner_map_fd));
|
||||
}
|
||||
|
||||
void bpf_gen__record_attach_target(struct bpf_gen *gen, const char *attach_name,
|
||||
enum bpf_attach_type type)
|
||||
{
|
||||
const char *prefix;
|
||||
int kind, ret;
|
||||
|
||||
btf_get_kernel_prefix_kind(type, &prefix, &kind);
|
||||
gen->attach_kind = kind;
|
||||
ret = snprintf(gen->attach_target, sizeof(gen->attach_target), "%s%s",
|
||||
prefix, attach_name);
|
||||
if (ret == sizeof(gen->attach_target))
|
||||
gen->error = -ENOSPC;
|
||||
}
|
||||
|
||||
static void emit_find_attach_target(struct bpf_gen *gen)
|
||||
{
|
||||
int name, len = strlen(gen->attach_target) + 1;
|
||||
|
||||
pr_debug("gen: find_attach_tgt %s %d\n", gen->attach_target, gen->attach_kind);
|
||||
name = add_data(gen, gen->attach_target, len);
|
||||
|
||||
emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE,
|
||||
0, 0, 0, name));
|
||||
emit(gen, BPF_MOV64_IMM(BPF_REG_2, len));
|
||||
emit(gen, BPF_MOV64_IMM(BPF_REG_3, gen->attach_kind));
|
||||
emit(gen, BPF_MOV64_IMM(BPF_REG_4, 0));
|
||||
emit(gen, BPF_EMIT_CALL(BPF_FUNC_btf_find_by_name_kind));
|
||||
emit(gen, BPF_MOV64_REG(BPF_REG_7, BPF_REG_0));
|
||||
debug_ret(gen, "find_by_name_kind(%s,%d)",
|
||||
gen->attach_target, gen->attach_kind);
|
||||
emit_check_err(gen);
|
||||
/* if successful, btf_id is in lower 32-bit of R7 and
|
||||
* btf_obj_fd is in upper 32-bit
|
||||
*/
|
||||
}
|
||||
|
||||
void bpf_gen__record_extern(struct bpf_gen *gen, const char *name, int kind,
|
||||
int insn_idx)
|
||||
{
|
||||
struct ksym_relo_desc *relo;
|
||||
|
||||
relo = libbpf_reallocarray(gen->relos, gen->relo_cnt + 1, sizeof(*relo));
|
||||
if (!relo) {
|
||||
gen->error = -ENOMEM;
|
||||
return;
|
||||
}
|
||||
gen->relos = relo;
|
||||
relo += gen->relo_cnt;
|
||||
relo->name = name;
|
||||
relo->kind = kind;
|
||||
relo->insn_idx = insn_idx;
|
||||
gen->relo_cnt++;
|
||||
}
|
||||
|
||||
static void emit_relo(struct bpf_gen *gen, struct ksym_relo_desc *relo, int insns)
|
||||
{
|
||||
int name, insn, len = strlen(relo->name) + 1;
|
||||
|
||||
pr_debug("gen: emit_relo: %s at %d\n", relo->name, relo->insn_idx);
|
||||
name = add_data(gen, relo->name, len);
|
||||
|
||||
emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE,
|
||||
0, 0, 0, name));
|
||||
emit(gen, BPF_MOV64_IMM(BPF_REG_2, len));
|
||||
emit(gen, BPF_MOV64_IMM(BPF_REG_3, relo->kind));
|
||||
emit(gen, BPF_MOV64_IMM(BPF_REG_4, 0));
|
||||
emit(gen, BPF_EMIT_CALL(BPF_FUNC_btf_find_by_name_kind));
|
||||
emit(gen, BPF_MOV64_REG(BPF_REG_7, BPF_REG_0));
|
||||
debug_ret(gen, "find_by_name_kind(%s,%d)", relo->name, relo->kind);
|
||||
emit_check_err(gen);
|
||||
/* store btf_id into insn[insn_idx].imm */
|
||||
insn = insns + sizeof(struct bpf_insn) * relo->insn_idx +
|
||||
offsetof(struct bpf_insn, imm);
|
||||
emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_0, BPF_PSEUDO_MAP_IDX_VALUE,
|
||||
0, 0, 0, insn));
|
||||
emit(gen, BPF_STX_MEM(BPF_W, BPF_REG_0, BPF_REG_7, 0));
|
||||
if (relo->kind == BTF_KIND_VAR) {
|
||||
/* store btf_obj_fd into insn[insn_idx + 1].imm */
|
||||
emit(gen, BPF_ALU64_IMM(BPF_RSH, BPF_REG_7, 32));
|
||||
emit(gen, BPF_STX_MEM(BPF_W, BPF_REG_0, BPF_REG_7,
|
||||
sizeof(struct bpf_insn)));
|
||||
}
|
||||
}
|
||||
|
||||
static void emit_relos(struct bpf_gen *gen, int insns)
|
||||
{
|
||||
int i;
|
||||
|
||||
for (i = 0; i < gen->relo_cnt; i++)
|
||||
emit_relo(gen, gen->relos + i, insns);
|
||||
}
|
||||
|
||||
static void cleanup_relos(struct bpf_gen *gen, int insns)
|
||||
{
|
||||
int i, insn;
|
||||
|
||||
for (i = 0; i < gen->relo_cnt; i++) {
|
||||
if (gen->relos[i].kind != BTF_KIND_VAR)
|
||||
continue;
|
||||
/* close fd recorded in insn[insn_idx + 1].imm */
|
||||
insn = insns +
|
||||
sizeof(struct bpf_insn) * (gen->relos[i].insn_idx + 1) +
|
||||
offsetof(struct bpf_insn, imm);
|
||||
emit_sys_close_blob(gen, insn);
|
||||
}
|
||||
if (gen->relo_cnt) {
|
||||
free(gen->relos);
|
||||
gen->relo_cnt = 0;
|
||||
gen->relos = NULL;
|
||||
}
|
||||
}
|
||||
|
||||
void bpf_gen__prog_load(struct bpf_gen *gen,
|
||||
struct bpf_prog_load_params *load_attr, int prog_idx)
|
||||
{
|
||||
int attr_size = offsetofend(union bpf_attr, fd_array);
|
||||
int prog_load_attr, license, insns, func_info, line_info;
|
||||
union bpf_attr attr;
|
||||
|
||||
memset(&attr, 0, attr_size);
|
||||
pr_debug("gen: prog_load: type %d insns_cnt %zd\n",
|
||||
load_attr->prog_type, load_attr->insn_cnt);
|
||||
/* add license string to blob of bytes */
|
||||
license = add_data(gen, load_attr->license, strlen(load_attr->license) + 1);
|
||||
/* add insns to blob of bytes */
|
||||
insns = add_data(gen, load_attr->insns,
|
||||
load_attr->insn_cnt * sizeof(struct bpf_insn));
|
||||
|
||||
attr.prog_type = load_attr->prog_type;
|
||||
attr.expected_attach_type = load_attr->expected_attach_type;
|
||||
attr.attach_btf_id = load_attr->attach_btf_id;
|
||||
attr.prog_ifindex = load_attr->prog_ifindex;
|
||||
attr.kern_version = 0;
|
||||
attr.insn_cnt = (__u32)load_attr->insn_cnt;
|
||||
attr.prog_flags = load_attr->prog_flags;
|
||||
|
||||
attr.func_info_rec_size = load_attr->func_info_rec_size;
|
||||
attr.func_info_cnt = load_attr->func_info_cnt;
|
||||
func_info = add_data(gen, load_attr->func_info,
|
||||
attr.func_info_cnt * attr.func_info_rec_size);
|
||||
|
||||
attr.line_info_rec_size = load_attr->line_info_rec_size;
|
||||
attr.line_info_cnt = load_attr->line_info_cnt;
|
||||
line_info = add_data(gen, load_attr->line_info,
|
||||
attr.line_info_cnt * attr.line_info_rec_size);
|
||||
|
||||
memcpy(attr.prog_name, load_attr->name,
|
||||
min((unsigned)strlen(load_attr->name), BPF_OBJ_NAME_LEN - 1));
|
||||
prog_load_attr = add_data(gen, &attr, attr_size);
|
||||
|
||||
/* populate union bpf_attr with a pointer to license */
|
||||
emit_rel_store(gen, attr_field(prog_load_attr, license), license);
|
||||
|
||||
/* populate union bpf_attr with a pointer to instructions */
|
||||
emit_rel_store(gen, attr_field(prog_load_attr, insns), insns);
|
||||
|
||||
/* populate union bpf_attr with a pointer to func_info */
|
||||
emit_rel_store(gen, attr_field(prog_load_attr, func_info), func_info);
|
||||
|
||||
/* populate union bpf_attr with a pointer to line_info */
|
||||
emit_rel_store(gen, attr_field(prog_load_attr, line_info), line_info);
|
||||
|
||||
/* populate union bpf_attr fd_array with a pointer to stack where map_fds are saved */
|
||||
emit_rel_store_sp(gen, attr_field(prog_load_attr, fd_array),
|
||||
stack_off(map_fd[0]));
|
||||
|
||||
/* populate union bpf_attr with user provided log details */
|
||||
move_ctx2blob(gen, attr_field(prog_load_attr, log_level), 4,
|
||||
offsetof(struct bpf_loader_ctx, log_level), false);
|
||||
move_ctx2blob(gen, attr_field(prog_load_attr, log_size), 4,
|
||||
offsetof(struct bpf_loader_ctx, log_size), false);
|
||||
move_ctx2blob(gen, attr_field(prog_load_attr, log_buf), 8,
|
||||
offsetof(struct bpf_loader_ctx, log_buf), false);
|
||||
/* populate union bpf_attr with btf_fd saved in the stack earlier */
|
||||
move_stack2blob(gen, attr_field(prog_load_attr, prog_btf_fd), 4,
|
||||
stack_off(btf_fd));
|
||||
if (gen->attach_kind) {
|
||||
emit_find_attach_target(gen);
|
||||
/* populate union bpf_attr with btf_id and btf_obj_fd found by helper */
|
||||
emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_0, BPF_PSEUDO_MAP_IDX_VALUE,
|
||||
0, 0, 0, prog_load_attr));
|
||||
emit(gen, BPF_STX_MEM(BPF_W, BPF_REG_0, BPF_REG_7,
|
||||
offsetof(union bpf_attr, attach_btf_id)));
|
||||
emit(gen, BPF_ALU64_IMM(BPF_RSH, BPF_REG_7, 32));
|
||||
emit(gen, BPF_STX_MEM(BPF_W, BPF_REG_0, BPF_REG_7,
|
||||
offsetof(union bpf_attr, attach_btf_obj_fd)));
|
||||
}
|
||||
emit_relos(gen, insns);
|
||||
/* emit PROG_LOAD command */
|
||||
emit_sys_bpf(gen, BPF_PROG_LOAD, prog_load_attr, attr_size);
|
||||
debug_ret(gen, "prog_load %s insn_cnt %d", attr.prog_name, attr.insn_cnt);
|
||||
/* successful or not, close btf module FDs used in extern ksyms and attach_btf_obj_fd */
|
||||
cleanup_relos(gen, insns);
|
||||
if (gen->attach_kind)
|
||||
emit_sys_close_blob(gen,
|
||||
attr_field(prog_load_attr, attach_btf_obj_fd));
|
||||
emit_check_err(gen);
|
||||
/* remember prog_fd in the stack, if successful */
|
||||
emit(gen, BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_7,
|
||||
stack_off(prog_fd[gen->nr_progs])));
|
||||
gen->nr_progs++;
|
||||
}
|
||||
|
||||
void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue,
|
||||
__u32 value_size)
|
||||
{
|
||||
int attr_size = offsetofend(union bpf_attr, flags);
|
||||
int map_update_attr, value, key;
|
||||
union bpf_attr attr;
|
||||
int zero = 0;
|
||||
|
||||
memset(&attr, 0, attr_size);
|
||||
pr_debug("gen: map_update_elem: idx %d\n", map_idx);
|
||||
|
||||
value = add_data(gen, pvalue, value_size);
|
||||
key = add_data(gen, &zero, sizeof(zero));
|
||||
|
||||
/* if (map_desc[map_idx].initial_value)
|
||||
* copy_from_user(value, initial_value, value_size);
|
||||
*/
|
||||
emit(gen, BPF_LDX_MEM(BPF_DW, BPF_REG_3, BPF_REG_6,
|
||||
sizeof(struct bpf_loader_ctx) +
|
||||
sizeof(struct bpf_map_desc) * map_idx +
|
||||
offsetof(struct bpf_map_desc, initial_value)));
|
||||
emit(gen, BPF_JMP_IMM(BPF_JEQ, BPF_REG_3, 0, 4));
|
||||
emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE,
|
||||
0, 0, 0, value));
|
||||
emit(gen, BPF_MOV64_IMM(BPF_REG_2, value_size));
|
||||
emit(gen, BPF_EMIT_CALL(BPF_FUNC_copy_from_user));
|
||||
|
||||
map_update_attr = add_data(gen, &attr, attr_size);
|
||||
move_stack2blob(gen, attr_field(map_update_attr, map_fd), 4,
|
||||
stack_off(map_fd[map_idx]));
|
||||
emit_rel_store(gen, attr_field(map_update_attr, key), key);
|
||||
emit_rel_store(gen, attr_field(map_update_attr, value), value);
|
||||
/* emit MAP_UPDATE_ELEM command */
|
||||
emit_sys_bpf(gen, BPF_MAP_UPDATE_ELEM, map_update_attr, attr_size);
|
||||
debug_ret(gen, "update_elem idx %d value_size %d", map_idx, value_size);
|
||||
emit_check_err(gen);
|
||||
}
|
||||
|
||||
void bpf_gen__map_freeze(struct bpf_gen *gen, int map_idx)
|
||||
{
|
||||
int attr_size = offsetofend(union bpf_attr, map_fd);
|
||||
int map_freeze_attr;
|
||||
union bpf_attr attr;
|
||||
|
||||
memset(&attr, 0, attr_size);
|
||||
pr_debug("gen: map_freeze: idx %d\n", map_idx);
|
||||
map_freeze_attr = add_data(gen, &attr, attr_size);
|
||||
move_stack2blob(gen, attr_field(map_freeze_attr, map_fd), 4,
|
||||
stack_off(map_fd[map_idx]));
|
||||
/* emit MAP_FREEZE command */
|
||||
emit_sys_bpf(gen, BPF_MAP_FREEZE, map_freeze_attr, attr_size);
|
||||
debug_ret(gen, "map_freeze");
|
||||
emit_check_err(gen);
|
||||
}
|
||||
@ -54,6 +54,7 @@
|
||||
#include "str_error.h"
|
||||
#include "libbpf_internal.h"
|
||||
#include "hashmap.h"
|
||||
#include "bpf_gen_internal.h"
|
||||
|
||||
#ifndef BPF_FS_MAGIC
|
||||
#define BPF_FS_MAGIC 0xcafe4a11
|
||||
@ -178,7 +179,7 @@ enum kern_feature_id {
|
||||
__FEAT_CNT,
|
||||
};
|
||||
|
||||
static bool kernel_supports(enum kern_feature_id feat_id);
|
||||
static bool kernel_supports(const struct bpf_object *obj, enum kern_feature_id feat_id);
|
||||
|
||||
enum reloc_type {
|
||||
RELO_LD64,
|
||||
@ -432,6 +433,8 @@ struct bpf_object {
|
||||
bool loaded;
|
||||
bool has_subcalls;
|
||||
|
||||
struct bpf_gen *gen_loader;
|
||||
|
||||
/*
|
||||
* Information when doing elf related work. Only valid if fd
|
||||
* is valid.
|
||||
@ -2463,20 +2466,20 @@ static bool section_have_execinstr(struct bpf_object *obj, int idx)
|
||||
|
||||
static bool btf_needs_sanitization(struct bpf_object *obj)
|
||||
{
|
||||
bool has_func_global = kernel_supports(FEAT_BTF_GLOBAL_FUNC);
|
||||
bool has_datasec = kernel_supports(FEAT_BTF_DATASEC);
|
||||
bool has_float = kernel_supports(FEAT_BTF_FLOAT);
|
||||
bool has_func = kernel_supports(FEAT_BTF_FUNC);
|
||||
bool has_func_global = kernel_supports(obj, FEAT_BTF_GLOBAL_FUNC);
|
||||
bool has_datasec = kernel_supports(obj, FEAT_BTF_DATASEC);
|
||||
bool has_float = kernel_supports(obj, FEAT_BTF_FLOAT);
|
||||
bool has_func = kernel_supports(obj, FEAT_BTF_FUNC);
|
||||
|
||||
return !has_func || !has_datasec || !has_func_global || !has_float;
|
||||
}
|
||||
|
||||
static void bpf_object__sanitize_btf(struct bpf_object *obj, struct btf *btf)
|
||||
{
|
||||
bool has_func_global = kernel_supports(FEAT_BTF_GLOBAL_FUNC);
|
||||
bool has_datasec = kernel_supports(FEAT_BTF_DATASEC);
|
||||
bool has_float = kernel_supports(FEAT_BTF_FLOAT);
|
||||
bool has_func = kernel_supports(FEAT_BTF_FUNC);
|
||||
bool has_func_global = kernel_supports(obj, FEAT_BTF_GLOBAL_FUNC);
|
||||
bool has_datasec = kernel_supports(obj, FEAT_BTF_DATASEC);
|
||||
bool has_float = kernel_supports(obj, FEAT_BTF_FLOAT);
|
||||
bool has_func = kernel_supports(obj, FEAT_BTF_FUNC);
|
||||
struct btf_type *t;
|
||||
int i, j, vlen;
|
||||
|
||||
@ -2657,7 +2660,7 @@ static int bpf_object__load_vmlinux_btf(struct bpf_object *obj, bool force)
|
||||
int err;
|
||||
|
||||
/* btf_vmlinux could be loaded earlier */
|
||||
if (obj->btf_vmlinux)
|
||||
if (obj->btf_vmlinux || obj->gen_loader)
|
||||
return 0;
|
||||
|
||||
if (!force && !obj_needs_vmlinux_btf(obj))
|
||||
@ -2682,7 +2685,7 @@ static int bpf_object__sanitize_and_load_btf(struct bpf_object *obj)
|
||||
if (!obj->btf)
|
||||
return 0;
|
||||
|
||||
if (!kernel_supports(FEAT_BTF)) {
|
||||
if (!kernel_supports(obj, FEAT_BTF)) {
|
||||
if (kernel_needs_btf(obj)) {
|
||||
err = -EOPNOTSUPP;
|
||||
goto report;
|
||||
@ -2739,7 +2742,20 @@ static int bpf_object__sanitize_and_load_btf(struct bpf_object *obj)
|
||||
bpf_object__sanitize_btf(obj, kern_btf);
|
||||
}
|
||||
|
||||
err = btf__load(kern_btf);
|
||||
if (obj->gen_loader) {
|
||||
__u32 raw_size = 0;
|
||||
const void *raw_data = btf__get_raw_data(kern_btf, &raw_size);
|
||||
|
||||
if (!raw_data)
|
||||
return -ENOMEM;
|
||||
bpf_gen__load_btf(obj->gen_loader, raw_data, raw_size);
|
||||
/* Pretend to have valid FD to pass various fd >= 0 checks.
|
||||
* This fd == 0 will not be used with any syscall and will be reset to -1 eventually.
|
||||
*/
|
||||
btf__set_fd(kern_btf, 0);
|
||||
} else {
|
||||
err = btf__load(kern_btf);
|
||||
}
|
||||
if (sanitize) {
|
||||
if (!err) {
|
||||
/* move fd to libbpf's BTF */
|
||||
@ -4310,11 +4326,17 @@ static struct kern_feature_desc {
|
||||
},
|
||||
};
|
||||
|
||||
static bool kernel_supports(enum kern_feature_id feat_id)
|
||||
static bool kernel_supports(const struct bpf_object *obj, enum kern_feature_id feat_id)
|
||||
{
|
||||
struct kern_feature_desc *feat = &feature_probes[feat_id];
|
||||
int ret;
|
||||
|
||||
if (obj->gen_loader)
|
||||
/* To generate loader program assume the latest kernel
|
||||
* to avoid doing extra prog_load, map_create syscalls.
|
||||
*/
|
||||
return true;
|
||||
|
||||
if (READ_ONCE(feat->res) == FEAT_UNKNOWN) {
|
||||
ret = feat->probe();
|
||||
if (ret > 0) {
|
||||
@ -4397,6 +4419,13 @@ bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map)
|
||||
char *cp, errmsg[STRERR_BUFSIZE];
|
||||
int err, zero = 0;
|
||||
|
||||
if (obj->gen_loader) {
|
||||
bpf_gen__map_update_elem(obj->gen_loader, map - obj->maps,
|
||||
map->mmaped, map->def.value_size);
|
||||
if (map_type == LIBBPF_MAP_RODATA || map_type == LIBBPF_MAP_KCONFIG)
|
||||
bpf_gen__map_freeze(obj->gen_loader, map - obj->maps);
|
||||
return 0;
|
||||
}
|
||||
err = bpf_map_update_elem(map->fd, &zero, map->mmaped, 0);
|
||||
if (err) {
|
||||
err = -errno;
|
||||
@ -4422,14 +4451,14 @@ bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map)
|
||||
|
||||
static void bpf_map__destroy(struct bpf_map *map);
|
||||
|
||||
static int bpf_object__create_map(struct bpf_object *obj, struct bpf_map *map)
|
||||
static int bpf_object__create_map(struct bpf_object *obj, struct bpf_map *map, bool is_inner)
|
||||
{
|
||||
struct bpf_create_map_attr create_attr;
|
||||
struct bpf_map_def *def = &map->def;
|
||||
|
||||
memset(&create_attr, 0, sizeof(create_attr));
|
||||
|
||||
if (kernel_supports(FEAT_PROG_NAME))
|
||||
if (kernel_supports(obj, FEAT_PROG_NAME))
|
||||
create_attr.name = map->name;
|
||||
create_attr.map_ifindex = map->map_ifindex;
|
||||
create_attr.map_type = def->type;
|
||||
@ -4470,7 +4499,7 @@ static int bpf_object__create_map(struct bpf_object *obj, struct bpf_map *map)
|
||||
if (map->inner_map) {
|
||||
int err;
|
||||
|
||||
err = bpf_object__create_map(obj, map->inner_map);
|
||||
err = bpf_object__create_map(obj, map->inner_map, true);
|
||||
if (err) {
|
||||
pr_warn("map '%s': failed to create inner map: %d\n",
|
||||
map->name, err);
|
||||
@ -4482,7 +4511,15 @@ static int bpf_object__create_map(struct bpf_object *obj, struct bpf_map *map)
|
||||
create_attr.inner_map_fd = map->inner_map_fd;
|
||||
}
|
||||
|
||||
map->fd = bpf_create_map_xattr(&create_attr);
|
||||
if (obj->gen_loader) {
|
||||
bpf_gen__map_create(obj->gen_loader, &create_attr, is_inner ? -1 : map - obj->maps);
|
||||
/* Pretend to have valid FD to pass various fd >= 0 checks.
|
||||
* This fd == 0 will not be used with any syscall and will be reset to -1 eventually.
|
||||
*/
|
||||
map->fd = 0;
|
||||
} else {
|
||||
map->fd = bpf_create_map_xattr(&create_attr);
|
||||
}
|
||||
if (map->fd < 0 && (create_attr.btf_key_type_id ||
|
||||
create_attr.btf_value_type_id)) {
|
||||
char *cp, errmsg[STRERR_BUFSIZE];
|
||||
@ -4503,6 +4540,8 @@ static int bpf_object__create_map(struct bpf_object *obj, struct bpf_map *map)
|
||||
return -errno;
|
||||
|
||||
if (bpf_map_type__is_map_in_map(def->type) && map->inner_map) {
|
||||
if (obj->gen_loader)
|
||||
map->inner_map->fd = -1;
|
||||
bpf_map__destroy(map->inner_map);
|
||||
zfree(&map->inner_map);
|
||||
}
|
||||
@ -4510,11 +4549,11 @@ static int bpf_object__create_map(struct bpf_object *obj, struct bpf_map *map)
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int init_map_slots(struct bpf_map *map)
|
||||
static int init_map_slots(struct bpf_object *obj, struct bpf_map *map)
|
||||
{
|
||||
const struct bpf_map *targ_map;
|
||||
unsigned int i;
|
||||
int fd, err;
|
||||
int fd, err = 0;
|
||||
|
||||
for (i = 0; i < map->init_slots_sz; i++) {
|
||||
if (!map->init_slots[i])
|
||||
@ -4522,7 +4561,13 @@ static int init_map_slots(struct bpf_map *map)
|
||||
|
||||
targ_map = map->init_slots[i];
|
||||
fd = bpf_map__fd(targ_map);
|
||||
err = bpf_map_update_elem(map->fd, &i, &fd, 0);
|
||||
if (obj->gen_loader) {
|
||||
pr_warn("// TODO map_update_elem: idx %ld key %d value==map_idx %ld\n",
|
||||
map - obj->maps, i, targ_map - obj->maps);
|
||||
return -ENOTSUP;
|
||||
} else {
|
||||
err = bpf_map_update_elem(map->fd, &i, &fd, 0);
|
||||
}
|
||||
if (err) {
|
||||
err = -errno;
|
||||
pr_warn("map '%s': failed to initialize slot [%d] to map '%s' fd=%d: %d\n",
|
||||
@ -4564,7 +4609,7 @@ bpf_object__create_maps(struct bpf_object *obj)
|
||||
pr_debug("map '%s': skipping creation (preset fd=%d)\n",
|
||||
map->name, map->fd);
|
||||
} else {
|
||||
err = bpf_object__create_map(obj, map);
|
||||
err = bpf_object__create_map(obj, map, false);
|
||||
if (err)
|
||||
goto err_out;
|
||||
|
||||
@ -4580,7 +4625,7 @@ bpf_object__create_maps(struct bpf_object *obj)
|
||||
}
|
||||
|
||||
if (map->init_slots_sz) {
|
||||
err = init_map_slots(map);
|
||||
err = init_map_slots(obj, map);
|
||||
if (err < 0) {
|
||||
zclose(map->fd);
|
||||
goto err_out;
|
||||
@ -4990,11 +5035,14 @@ static int load_module_btfs(struct bpf_object *obj)
|
||||
if (obj->btf_modules_loaded)
|
||||
return 0;
|
||||
|
||||
if (obj->gen_loader)
|
||||
return 0;
|
||||
|
||||
/* don't do this again, even if we find no module BTFs */
|
||||
obj->btf_modules_loaded = true;
|
||||
|
||||
/* kernel too old to support module BTFs */
|
||||
if (!kernel_supports(FEAT_MODULE_BTF))
|
||||
if (!kernel_supports(obj, FEAT_MODULE_BTF))
|
||||
return 0;
|
||||
|
||||
while (true) {
|
||||
@ -6137,6 +6185,12 @@ static int bpf_core_apply_relo(struct bpf_program *prog,
|
||||
if (str_is_empty(spec_str))
|
||||
return -EINVAL;
|
||||
|
||||
if (prog->obj->gen_loader) {
|
||||
pr_warn("// TODO core_relo: prog %ld insn[%d] %s %s kind %d\n",
|
||||
prog - prog->obj->programs, relo->insn_off / 8,
|
||||
local_name, spec_str, relo->kind);
|
||||
return -ENOTSUP;
|
||||
}
|
||||
err = bpf_core_parse_spec(local_btf, local_id, spec_str, relo->kind, &local_spec);
|
||||
if (err) {
|
||||
pr_warn("prog '%s': relo #%d: parsing [%d] %s %s + %s failed: %d\n",
|
||||
@ -6388,19 +6442,34 @@ bpf_object__relocate_data(struct bpf_object *obj, struct bpf_program *prog)
|
||||
|
||||
switch (relo->type) {
|
||||
case RELO_LD64:
|
||||
insn[0].src_reg = BPF_PSEUDO_MAP_FD;
|
||||
insn[0].imm = obj->maps[relo->map_idx].fd;
|
||||
if (obj->gen_loader) {
|
||||
insn[0].src_reg = BPF_PSEUDO_MAP_IDX;
|
||||
insn[0].imm = relo->map_idx;
|
||||
} else {
|
||||
insn[0].src_reg = BPF_PSEUDO_MAP_FD;
|
||||
insn[0].imm = obj->maps[relo->map_idx].fd;
|
||||
}
|
||||
break;
|
||||
case RELO_DATA:
|
||||
insn[0].src_reg = BPF_PSEUDO_MAP_VALUE;
|
||||
insn[1].imm = insn[0].imm + relo->sym_off;
|
||||
insn[0].imm = obj->maps[relo->map_idx].fd;
|
||||
if (obj->gen_loader) {
|
||||
insn[0].src_reg = BPF_PSEUDO_MAP_IDX_VALUE;
|
||||
insn[0].imm = relo->map_idx;
|
||||
} else {
|
||||
insn[0].src_reg = BPF_PSEUDO_MAP_VALUE;
|
||||
insn[0].imm = obj->maps[relo->map_idx].fd;
|
||||
}
|
||||
break;
|
||||
case RELO_EXTERN_VAR:
|
||||
ext = &obj->externs[relo->sym_off];
|
||||
if (ext->type == EXT_KCFG) {
|
||||
insn[0].src_reg = BPF_PSEUDO_MAP_VALUE;
|
||||
insn[0].imm = obj->maps[obj->kconfig_map_idx].fd;
|
||||
if (obj->gen_loader) {
|
||||
insn[0].src_reg = BPF_PSEUDO_MAP_IDX_VALUE;
|
||||
insn[0].imm = obj->kconfig_map_idx;
|
||||
} else {
|
||||
insn[0].src_reg = BPF_PSEUDO_MAP_VALUE;
|
||||
insn[0].imm = obj->maps[obj->kconfig_map_idx].fd;
|
||||
}
|
||||
insn[1].imm = ext->kcfg.data_off;
|
||||
} else /* EXT_KSYM */ {
|
||||
if (ext->ksym.type_id) { /* typed ksyms */
|
||||
@ -6419,11 +6488,15 @@ bpf_object__relocate_data(struct bpf_object *obj, struct bpf_program *prog)
|
||||
insn[0].imm = ext->ksym.kernel_btf_id;
|
||||
break;
|
||||
case RELO_SUBPROG_ADDR:
|
||||
insn[0].src_reg = BPF_PSEUDO_FUNC;
|
||||
/* will be handled as a follow up pass */
|
||||
if (insn[0].src_reg != BPF_PSEUDO_FUNC) {
|
||||
pr_warn("prog '%s': relo #%d: bad insn\n",
|
||||
prog->name, i);
|
||||
return -EINVAL;
|
||||
}
|
||||
/* handled already */
|
||||
break;
|
||||
case RELO_CALL:
|
||||
/* will be handled as a follow up pass */
|
||||
/* handled already */
|
||||
break;
|
||||
default:
|
||||
pr_warn("prog '%s': relo #%d: bad relo type %d\n",
|
||||
@ -6514,7 +6587,7 @@ reloc_prog_func_and_line_info(const struct bpf_object *obj,
|
||||
/* no .BTF.ext relocation if .BTF.ext is missing or kernel doesn't
|
||||
* supprot func/line info
|
||||
*/
|
||||
if (!obj->btf_ext || !kernel_supports(FEAT_BTF_FUNC))
|
||||
if (!obj->btf_ext || !kernel_supports(obj, FEAT_BTF_FUNC))
|
||||
return 0;
|
||||
|
||||
/* only attempt func info relocation if main program's func_info
|
||||
@ -6592,6 +6665,30 @@ static struct reloc_desc *find_prog_insn_relo(const struct bpf_program *prog, si
|
||||
sizeof(*prog->reloc_desc), cmp_relo_by_insn_idx);
|
||||
}
|
||||
|
||||
static int append_subprog_relos(struct bpf_program *main_prog, struct bpf_program *subprog)
|
||||
{
|
||||
int new_cnt = main_prog->nr_reloc + subprog->nr_reloc;
|
||||
struct reloc_desc *relos;
|
||||
int i;
|
||||
|
||||
if (main_prog == subprog)
|
||||
return 0;
|
||||
relos = libbpf_reallocarray(main_prog->reloc_desc, new_cnt, sizeof(*relos));
|
||||
if (!relos)
|
||||
return -ENOMEM;
|
||||
memcpy(relos + main_prog->nr_reloc, subprog->reloc_desc,
|
||||
sizeof(*relos) * subprog->nr_reloc);
|
||||
|
||||
for (i = main_prog->nr_reloc; i < new_cnt; i++)
|
||||
relos[i].insn_idx += subprog->sub_insn_off;
|
||||
/* After insn_idx adjustment the 'relos' array is still sorted
|
||||
* by insn_idx and doesn't break bsearch.
|
||||
*/
|
||||
main_prog->reloc_desc = relos;
|
||||
main_prog->nr_reloc = new_cnt;
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int
|
||||
bpf_object__reloc_code(struct bpf_object *obj, struct bpf_program *main_prog,
|
||||
struct bpf_program *prog)
|
||||
@ -6612,6 +6709,11 @@ bpf_object__reloc_code(struct bpf_object *obj, struct bpf_program *main_prog,
|
||||
continue;
|
||||
|
||||
relo = find_prog_insn_relo(prog, insn_idx);
|
||||
if (relo && relo->type == RELO_EXTERN_FUNC)
|
||||
/* kfunc relocations will be handled later
|
||||
* in bpf_object__relocate_data()
|
||||
*/
|
||||
continue;
|
||||
if (relo && relo->type != RELO_CALL && relo->type != RELO_SUBPROG_ADDR) {
|
||||
pr_warn("prog '%s': unexpected relo for insn #%zu, type %d\n",
|
||||
prog->name, insn_idx, relo->type);
|
||||
@ -6686,6 +6788,10 @@ bpf_object__reloc_code(struct bpf_object *obj, struct bpf_program *main_prog,
|
||||
pr_debug("prog '%s': added %zu insns from sub-prog '%s'\n",
|
||||
main_prog->name, subprog->insns_cnt, subprog->name);
|
||||
|
||||
/* The subprog insns are now appended. Append its relos too. */
|
||||
err = append_subprog_relos(main_prog, subprog);
|
||||
if (err)
|
||||
return err;
|
||||
err = bpf_object__reloc_code(obj, main_prog, subprog);
|
||||
if (err)
|
||||
return err;
|
||||
@ -6815,11 +6921,25 @@ bpf_object__relocate_calls(struct bpf_object *obj, struct bpf_program *prog)
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void
|
||||
bpf_object__free_relocs(struct bpf_object *obj)
|
||||
{
|
||||
struct bpf_program *prog;
|
||||
int i;
|
||||
|
||||
/* free up relocation descriptors */
|
||||
for (i = 0; i < obj->nr_programs; i++) {
|
||||
prog = &obj->programs[i];
|
||||
zfree(&prog->reloc_desc);
|
||||
prog->nr_reloc = 0;
|
||||
}
|
||||
}
|
||||
|
||||
static int
|
||||
bpf_object__relocate(struct bpf_object *obj, const char *targ_btf_path)
|
||||
{
|
||||
struct bpf_program *prog;
|
||||
size_t i;
|
||||
size_t i, j;
|
||||
int err;
|
||||
|
||||
if (obj->btf_ext) {
|
||||
@ -6830,23 +6950,32 @@ bpf_object__relocate(struct bpf_object *obj, const char *targ_btf_path)
|
||||
return err;
|
||||
}
|
||||
}
|
||||
/* relocate data references first for all programs and sub-programs,
|
||||
* as they don't change relative to code locations, so subsequent
|
||||
* subprogram processing won't need to re-calculate any of them
|
||||
|
||||
/* Before relocating calls pre-process relocations and mark
|
||||
* few ld_imm64 instructions that points to subprogs.
|
||||
* Otherwise bpf_object__reloc_code() later would have to consider
|
||||
* all ld_imm64 insns as relocation candidates. That would
|
||||
* reduce relocation speed, since amount of find_prog_insn_relo()
|
||||
* would increase and most of them will fail to find a relo.
|
||||
*/
|
||||
for (i = 0; i < obj->nr_programs; i++) {
|
||||
prog = &obj->programs[i];
|
||||
err = bpf_object__relocate_data(obj, prog);
|
||||
if (err) {
|
||||
pr_warn("prog '%s': failed to relocate data references: %d\n",
|
||||
prog->name, err);
|
||||
return err;
|
||||
for (j = 0; j < prog->nr_reloc; j++) {
|
||||
struct reloc_desc *relo = &prog->reloc_desc[j];
|
||||
struct bpf_insn *insn = &prog->insns[relo->insn_idx];
|
||||
|
||||
/* mark the insn, so it's recognized by insn_is_pseudo_func() */
|
||||
if (relo->type == RELO_SUBPROG_ADDR)
|
||||
insn[0].src_reg = BPF_PSEUDO_FUNC;
|
||||
}
|
||||
}
|
||||
/* now relocate subprogram calls and append used subprograms to main
|
||||
|
||||
/* relocate subprogram calls and append used subprograms to main
|
||||
* programs; each copy of subprogram code needs to be relocated
|
||||
* differently for each main program, because its code location might
|
||||
* have changed
|
||||
* have changed.
|
||||
* Append subprog relos to main programs to allow data relos to be
|
||||
* processed after text is completely relocated.
|
||||
*/
|
||||
for (i = 0; i < obj->nr_programs; i++) {
|
||||
prog = &obj->programs[i];
|
||||
@ -6863,12 +6992,20 @@ bpf_object__relocate(struct bpf_object *obj, const char *targ_btf_path)
|
||||
return err;
|
||||
}
|
||||
}
|
||||
/* free up relocation descriptors */
|
||||
/* Process data relos for main programs */
|
||||
for (i = 0; i < obj->nr_programs; i++) {
|
||||
prog = &obj->programs[i];
|
||||
zfree(&prog->reloc_desc);
|
||||
prog->nr_reloc = 0;
|
||||
if (prog_is_subprog(obj, prog))
|
||||
continue;
|
||||
err = bpf_object__relocate_data(obj, prog);
|
||||
if (err) {
|
||||
pr_warn("prog '%s': failed to relocate data references: %d\n",
|
||||
prog->name, err);
|
||||
return err;
|
||||
}
|
||||
}
|
||||
if (!obj->gen_loader)
|
||||
bpf_object__free_relocs(obj);
|
||||
return 0;
|
||||
}
|
||||
|
||||
@ -7057,6 +7194,9 @@ static int bpf_object__sanitize_prog(struct bpf_object *obj, struct bpf_program
|
||||
enum bpf_func_id func_id;
|
||||
int i;
|
||||
|
||||
if (obj->gen_loader)
|
||||
return 0;
|
||||
|
||||
for (i = 0; i < prog->insns_cnt; i++, insn++) {
|
||||
if (!insn_is_helper_call(insn, &func_id))
|
||||
continue;
|
||||
@ -7068,12 +7208,12 @@ static int bpf_object__sanitize_prog(struct bpf_object *obj, struct bpf_program
|
||||
switch (func_id) {
|
||||
case BPF_FUNC_probe_read_kernel:
|
||||
case BPF_FUNC_probe_read_user:
|
||||
if (!kernel_supports(FEAT_PROBE_READ_KERN))
|
||||
if (!kernel_supports(obj, FEAT_PROBE_READ_KERN))
|
||||
insn->imm = BPF_FUNC_probe_read;
|
||||
break;
|
||||
case BPF_FUNC_probe_read_kernel_str:
|
||||
case BPF_FUNC_probe_read_user_str:
|
||||
if (!kernel_supports(FEAT_PROBE_READ_KERN))
|
||||
if (!kernel_supports(obj, FEAT_PROBE_READ_KERN))
|
||||
insn->imm = BPF_FUNC_probe_read_str;
|
||||
break;
|
||||
default:
|
||||
@ -7108,12 +7248,12 @@ load_program(struct bpf_program *prog, struct bpf_insn *insns, int insns_cnt,
|
||||
|
||||
load_attr.prog_type = prog->type;
|
||||
/* old kernels might not support specifying expected_attach_type */
|
||||
if (!kernel_supports(FEAT_EXP_ATTACH_TYPE) && prog->sec_def &&
|
||||
if (!kernel_supports(prog->obj, FEAT_EXP_ATTACH_TYPE) && prog->sec_def &&
|
||||
prog->sec_def->is_exp_attach_type_optional)
|
||||
load_attr.expected_attach_type = 0;
|
||||
else
|
||||
load_attr.expected_attach_type = prog->expected_attach_type;
|
||||
if (kernel_supports(FEAT_PROG_NAME))
|
||||
if (kernel_supports(prog->obj, FEAT_PROG_NAME))
|
||||
load_attr.name = prog->name;
|
||||
load_attr.insns = insns;
|
||||
load_attr.insn_cnt = insns_cnt;
|
||||
@ -7129,7 +7269,7 @@ load_program(struct bpf_program *prog, struct bpf_insn *insns, int insns_cnt,
|
||||
|
||||
/* specify func_info/line_info only if kernel supports them */
|
||||
btf_fd = bpf_object__btf_fd(prog->obj);
|
||||
if (btf_fd >= 0 && kernel_supports(FEAT_BTF_FUNC)) {
|
||||
if (btf_fd >= 0 && kernel_supports(prog->obj, FEAT_BTF_FUNC)) {
|
||||
load_attr.prog_btf_fd = btf_fd;
|
||||
load_attr.func_info = prog->func_info;
|
||||
load_attr.func_info_rec_size = prog->func_info_rec_size;
|
||||
@ -7141,6 +7281,12 @@ load_program(struct bpf_program *prog, struct bpf_insn *insns, int insns_cnt,
|
||||
load_attr.log_level = prog->log_level;
|
||||
load_attr.prog_flags = prog->prog_flags;
|
||||
|
||||
if (prog->obj->gen_loader) {
|
||||
bpf_gen__prog_load(prog->obj->gen_loader, &load_attr,
|
||||
prog - prog->obj->programs);
|
||||
*pfd = -1;
|
||||
return 0;
|
||||
}
|
||||
retry_load:
|
||||
if (log_buf_size) {
|
||||
log_buf = malloc(log_buf_size);
|
||||
@ -7159,7 +7305,7 @@ retry_load:
|
||||
pr_debug("verifier log:\n%s", log_buf);
|
||||
|
||||
if (prog->obj->rodata_map_idx >= 0 &&
|
||||
kernel_supports(FEAT_PROG_BIND_MAP)) {
|
||||
kernel_supports(prog->obj, FEAT_PROG_BIND_MAP)) {
|
||||
struct bpf_map *rodata_map =
|
||||
&prog->obj->maps[prog->obj->rodata_map_idx];
|
||||
|
||||
@ -7218,6 +7364,38 @@ out:
|
||||
return ret;
|
||||
}
|
||||
|
||||
static int bpf_program__record_externs(struct bpf_program *prog)
|
||||
{
|
||||
struct bpf_object *obj = prog->obj;
|
||||
int i;
|
||||
|
||||
for (i = 0; i < prog->nr_reloc; i++) {
|
||||
struct reloc_desc *relo = &prog->reloc_desc[i];
|
||||
struct extern_desc *ext = &obj->externs[relo->sym_off];
|
||||
|
||||
switch (relo->type) {
|
||||
case RELO_EXTERN_VAR:
|
||||
if (ext->type != EXT_KSYM)
|
||||
continue;
|
||||
if (!ext->ksym.type_id) {
|
||||
pr_warn("typeless ksym %s is not supported yet\n",
|
||||
ext->name);
|
||||
return -ENOTSUP;
|
||||
}
|
||||
bpf_gen__record_extern(obj->gen_loader, ext->name, BTF_KIND_VAR,
|
||||
relo->insn_idx);
|
||||
break;
|
||||
case RELO_EXTERN_FUNC:
|
||||
bpf_gen__record_extern(obj->gen_loader, ext->name, BTF_KIND_FUNC,
|
||||
relo->insn_idx);
|
||||
break;
|
||||
default:
|
||||
continue;
|
||||
}
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int libbpf_find_attach_btf_id(struct bpf_program *prog, int *btf_obj_fd, int *btf_type_id);
|
||||
|
||||
int bpf_program__load(struct bpf_program *prog, char *license, __u32 kern_ver)
|
||||
@ -7263,6 +7441,8 @@ int bpf_program__load(struct bpf_program *prog, char *license, __u32 kern_ver)
|
||||
pr_warn("prog '%s': inconsistent nr(%d) != 1\n",
|
||||
prog->name, prog->instances.nr);
|
||||
}
|
||||
if (prog->obj->gen_loader)
|
||||
bpf_program__record_externs(prog);
|
||||
err = load_program(prog, prog->insns, prog->insns_cnt,
|
||||
license, kern_ver, &fd);
|
||||
if (!err)
|
||||
@ -7339,6 +7519,8 @@ bpf_object__load_progs(struct bpf_object *obj, int log_level)
|
||||
if (err)
|
||||
return err;
|
||||
}
|
||||
if (obj->gen_loader)
|
||||
bpf_object__free_relocs(obj);
|
||||
return 0;
|
||||
}
|
||||
|
||||
@ -7517,11 +7699,11 @@ static int bpf_object__sanitize_maps(struct bpf_object *obj)
|
||||
bpf_object__for_each_map(m, obj) {
|
||||
if (!bpf_map__is_internal(m))
|
||||
continue;
|
||||
if (!kernel_supports(FEAT_GLOBAL_DATA)) {
|
||||
if (!kernel_supports(obj, FEAT_GLOBAL_DATA)) {
|
||||
pr_warn("kernel doesn't support global data\n");
|
||||
return -ENOTSUP;
|
||||
}
|
||||
if (!kernel_supports(FEAT_ARRAY_MMAP))
|
||||
if (!kernel_supports(obj, FEAT_ARRAY_MMAP))
|
||||
m->def.map_flags ^= BPF_F_MMAPABLE;
|
||||
}
|
||||
|
||||
@ -7719,6 +7901,12 @@ static int bpf_object__resolve_ksyms_btf_id(struct bpf_object *obj)
|
||||
if (ext->type != EXT_KSYM || !ext->ksym.type_id)
|
||||
continue;
|
||||
|
||||
if (obj->gen_loader) {
|
||||
ext->is_set = true;
|
||||
ext->ksym.kernel_btf_obj_fd = 0;
|
||||
ext->ksym.kernel_btf_id = 0;
|
||||
continue;
|
||||
}
|
||||
t = btf__type_by_id(obj->btf, ext->btf_id);
|
||||
if (btf_is_var(t))
|
||||
err = bpf_object__resolve_ksym_var_btf_id(obj, ext);
|
||||
@ -7833,6 +8021,9 @@ int bpf_object__load_xattr(struct bpf_object_load_attr *attr)
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
if (obj->gen_loader)
|
||||
bpf_gen__init(obj->gen_loader, attr->log_level);
|
||||
|
||||
err = bpf_object__probe_loading(obj);
|
||||
err = err ? : bpf_object__load_vmlinux_btf(obj, false);
|
||||
err = err ? : bpf_object__resolve_externs(obj, obj->kconfig);
|
||||
@ -7843,6 +8034,15 @@ int bpf_object__load_xattr(struct bpf_object_load_attr *attr)
|
||||
err = err ? : bpf_object__relocate(obj, attr->target_btf_path);
|
||||
err = err ? : bpf_object__load_progs(obj, attr->log_level);
|
||||
|
||||
if (obj->gen_loader) {
|
||||
/* reset FDs */
|
||||
btf__set_fd(obj->btf, -1);
|
||||
for (i = 0; i < obj->nr_maps; i++)
|
||||
obj->maps[i].fd = -1;
|
||||
if (!err)
|
||||
err = bpf_gen__finish(obj->gen_loader);
|
||||
}
|
||||
|
||||
/* clean up module BTFs */
|
||||
for (i = 0; i < obj->btf_module_cnt; i++) {
|
||||
close(obj->btf_modules[i].fd);
|
||||
@ -8468,6 +8668,7 @@ void bpf_object__close(struct bpf_object *obj)
|
||||
if (obj->clear_priv)
|
||||
obj->clear_priv(obj, obj->priv);
|
||||
|
||||
bpf_gen__free(obj->gen_loader);
|
||||
bpf_object__elf_finish(obj);
|
||||
bpf_object__unload(obj);
|
||||
btf__free(obj->btf);
|
||||
@ -8558,6 +8759,22 @@ void *bpf_object__priv(const struct bpf_object *obj)
|
||||
return obj ? obj->priv : ERR_PTR(-EINVAL);
|
||||
}
|
||||
|
||||
int bpf_object__gen_loader(struct bpf_object *obj, struct gen_loader_opts *opts)
|
||||
{
|
||||
struct bpf_gen *gen;
|
||||
|
||||
if (!opts)
|
||||
return -EFAULT;
|
||||
if (!OPTS_VALID(opts, gen_loader_opts))
|
||||
return -EINVAL;
|
||||
gen = calloc(sizeof(*gen), 1);
|
||||
if (!gen)
|
||||
return -ENOMEM;
|
||||
gen->opts = opts;
|
||||
obj->gen_loader = gen;
|
||||
return 0;
|
||||
}
|
||||
|
||||
static struct bpf_program *
|
||||
__bpf_program__iter(const struct bpf_program *p, const struct bpf_object *obj,
|
||||
bool forward)
|
||||
@ -8904,6 +9121,8 @@ static const struct bpf_sec_def section_defs[] = {
|
||||
.expected_attach_type = BPF_TRACE_ITER,
|
||||
.is_attach_btf = true,
|
||||
.attach_fn = attach_iter),
|
||||
SEC_DEF("syscall", SYSCALL,
|
||||
.is_sleepable = true),
|
||||
BPF_EAPROG_SEC("xdp_devmap/", BPF_PROG_TYPE_XDP,
|
||||
BPF_XDP_DEVMAP),
|
||||
BPF_EAPROG_SEC("xdp_cpumap/", BPF_PROG_TYPE_XDP,
|
||||
@ -9193,6 +9412,28 @@ invalid_prog:
|
||||
#define BTF_ITER_PREFIX "bpf_iter_"
|
||||
#define BTF_MAX_NAME_SIZE 128
|
||||
|
||||
void btf_get_kernel_prefix_kind(enum bpf_attach_type attach_type,
|
||||
const char **prefix, int *kind)
|
||||
{
|
||||
switch (attach_type) {
|
||||
case BPF_TRACE_RAW_TP:
|
||||
*prefix = BTF_TRACE_PREFIX;
|
||||
*kind = BTF_KIND_TYPEDEF;
|
||||
break;
|
||||
case BPF_LSM_MAC:
|
||||
*prefix = BTF_LSM_PREFIX;
|
||||
*kind = BTF_KIND_FUNC;
|
||||
break;
|
||||
case BPF_TRACE_ITER:
|
||||
*prefix = BTF_ITER_PREFIX;
|
||||
*kind = BTF_KIND_FUNC;
|
||||
break;
|
||||
default:
|
||||
*prefix = "";
|
||||
*kind = BTF_KIND_FUNC;
|
||||
}
|
||||
}
|
||||
|
||||
static int find_btf_by_prefix_kind(const struct btf *btf, const char *prefix,
|
||||
const char *name, __u32 kind)
|
||||
{
|
||||
@ -9213,21 +9454,11 @@ static int find_btf_by_prefix_kind(const struct btf *btf, const char *prefix,
|
||||
static inline int find_attach_btf_id(struct btf *btf, const char *name,
|
||||
enum bpf_attach_type attach_type)
|
||||
{
|
||||
int err;
|
||||
const char *prefix;
|
||||
int kind;
|
||||
|
||||
if (attach_type == BPF_TRACE_RAW_TP)
|
||||
err = find_btf_by_prefix_kind(btf, BTF_TRACE_PREFIX, name,
|
||||
BTF_KIND_TYPEDEF);
|
||||
else if (attach_type == BPF_LSM_MAC)
|
||||
err = find_btf_by_prefix_kind(btf, BTF_LSM_PREFIX, name,
|
||||
BTF_KIND_FUNC);
|
||||
else if (attach_type == BPF_TRACE_ITER)
|
||||
err = find_btf_by_prefix_kind(btf, BTF_ITER_PREFIX, name,
|
||||
BTF_KIND_FUNC);
|
||||
else
|
||||
err = btf__find_by_name_kind(btf, name, BTF_KIND_FUNC);
|
||||
|
||||
return err;
|
||||
btf_get_kernel_prefix_kind(attach_type, &prefix, &kind);
|
||||
return find_btf_by_prefix_kind(btf, prefix, name, kind);
|
||||
}
|
||||
|
||||
int libbpf_find_vmlinux_btf_id(const char *name,
|
||||
@ -9326,7 +9557,7 @@ static int libbpf_find_attach_btf_id(struct bpf_program *prog, int *btf_obj_fd,
|
||||
__u32 attach_prog_fd = prog->attach_prog_fd;
|
||||
const char *name = prog->sec_name, *attach_name;
|
||||
const struct bpf_sec_def *sec = NULL;
|
||||
int i, err;
|
||||
int i, err = 0;
|
||||
|
||||
if (!name)
|
||||
return -EINVAL;
|
||||
@ -9361,7 +9592,13 @@ static int libbpf_find_attach_btf_id(struct bpf_program *prog, int *btf_obj_fd,
|
||||
}
|
||||
|
||||
/* kernel/module BTF ID */
|
||||
err = find_kernel_btf_id(prog->obj, attach_name, attach_type, btf_obj_fd, btf_type_id);
|
||||
if (prog->obj->gen_loader) {
|
||||
bpf_gen__record_attach_target(prog->obj->gen_loader, attach_name, attach_type);
|
||||
*btf_obj_fd = 0;
|
||||
*btf_type_id = 1;
|
||||
} else {
|
||||
err = find_kernel_btf_id(prog->obj, attach_name, attach_type, btf_obj_fd, btf_type_id);
|
||||
}
|
||||
if (err) {
|
||||
pr_warn("failed to find kernel BTF type ID of '%s': %d\n", attach_name, err);
|
||||
return err;
|
||||
@ -9518,6 +9755,14 @@ int bpf_map__set_initial_value(struct bpf_map *map,
|
||||
return 0;
|
||||
}
|
||||
|
||||
const void *bpf_map__initial_value(struct bpf_map *map, size_t *psize)
|
||||
{
|
||||
if (!map->mmaped)
|
||||
return NULL;
|
||||
*psize = map->def.value_size;
|
||||
return map->mmaped;
|
||||
}
|
||||
|
||||
bool bpf_map__is_offload_neutral(const struct bpf_map *map)
|
||||
{
|
||||
return map->def.type == BPF_MAP_TYPE_PERF_EVENT_ARRAY;
|
||||
|
||||
@ -471,6 +471,7 @@ LIBBPF_API int bpf_map__set_priv(struct bpf_map *map, void *priv,
|
||||
LIBBPF_API void *bpf_map__priv(const struct bpf_map *map);
|
||||
LIBBPF_API int bpf_map__set_initial_value(struct bpf_map *map,
|
||||
const void *data, size_t size);
|
||||
LIBBPF_API const void *bpf_map__initial_value(struct bpf_map *map, size_t *psize);
|
||||
LIBBPF_API bool bpf_map__is_offload_neutral(const struct bpf_map *map);
|
||||
LIBBPF_API bool bpf_map__is_internal(const struct bpf_map *map);
|
||||
LIBBPF_API int bpf_map__set_pin_path(struct bpf_map *map, const char *path);
|
||||
@ -800,6 +801,18 @@ LIBBPF_API int bpf_object__attach_skeleton(struct bpf_object_skeleton *s);
|
||||
LIBBPF_API void bpf_object__detach_skeleton(struct bpf_object_skeleton *s);
|
||||
LIBBPF_API void bpf_object__destroy_skeleton(struct bpf_object_skeleton *s);
|
||||
|
||||
struct gen_loader_opts {
|
||||
size_t sz; /* size of this struct, for forward/backward compatiblity */
|
||||
const char *data;
|
||||
const char *insns;
|
||||
__u32 data_sz;
|
||||
__u32 insns_sz;
|
||||
};
|
||||
|
||||
#define gen_loader_opts__last_field insns_sz
|
||||
LIBBPF_API int bpf_object__gen_loader(struct bpf_object *obj,
|
||||
struct gen_loader_opts *opts);
|
||||
|
||||
enum libbpf_tristate {
|
||||
TRI_NO = 0,
|
||||
TRI_YES = 1,
|
||||
|
||||
@ -359,7 +359,9 @@ LIBBPF_0.4.0 {
|
||||
bpf_linker__finalize;
|
||||
bpf_linker__free;
|
||||
bpf_linker__new;
|
||||
bpf_map__initial_value;
|
||||
bpf_map__inner_map;
|
||||
bpf_object__gen_loader;
|
||||
bpf_object__set_kversion;
|
||||
bpf_tc_attach;
|
||||
bpf_tc_detach;
|
||||
|
||||
@ -258,6 +258,8 @@ int bpf_object__section_size(const struct bpf_object *obj, const char *name,
|
||||
int bpf_object__variable_offset(const struct bpf_object *obj, const char *name,
|
||||
__u32 *off);
|
||||
struct btf *btf_get_from_fd(int btf_fd, struct btf *base_btf);
|
||||
void btf_get_kernel_prefix_kind(enum bpf_attach_type attach_type,
|
||||
const char **prefix, int *kind);
|
||||
|
||||
struct btf_ext_info {
|
||||
/*
|
||||
|
||||
123
tools/lib/bpf/skel_internal.h
Normal file
123
tools/lib/bpf/skel_internal.h
Normal file
@ -0,0 +1,123 @@
|
||||
/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */
|
||||
/* Copyright (c) 2021 Facebook */
|
||||
#ifndef __SKEL_INTERNAL_H
|
||||
#define __SKEL_INTERNAL_H
|
||||
|
||||
#include <unistd.h>
|
||||
#include <sys/syscall.h>
|
||||
#include <sys/mman.h>
|
||||
|
||||
/* This file is a base header for auto-generated *.lskel.h files.
|
||||
* Its contents will change and may become part of auto-generation in the future.
|
||||
*
|
||||
* The layout of bpf_[map|prog]_desc and bpf_loader_ctx is feature dependent
|
||||
* and will change from one version of libbpf to another and features
|
||||
* requested during loader program generation.
|
||||
*/
|
||||
struct bpf_map_desc {
|
||||
union {
|
||||
/* input for the loader prog */
|
||||
struct {
|
||||
__aligned_u64 initial_value;
|
||||
__u32 max_entries;
|
||||
};
|
||||
/* output of the loader prog */
|
||||
struct {
|
||||
int map_fd;
|
||||
};
|
||||
};
|
||||
};
|
||||
struct bpf_prog_desc {
|
||||
int prog_fd;
|
||||
};
|
||||
|
||||
struct bpf_loader_ctx {
|
||||
size_t sz;
|
||||
__u32 log_level;
|
||||
__u32 log_size;
|
||||
__u64 log_buf;
|
||||
};
|
||||
|
||||
struct bpf_load_and_run_opts {
|
||||
struct bpf_loader_ctx *ctx;
|
||||
const void *data;
|
||||
const void *insns;
|
||||
__u32 data_sz;
|
||||
__u32 insns_sz;
|
||||
const char *errstr;
|
||||
};
|
||||
|
||||
static inline int skel_sys_bpf(enum bpf_cmd cmd, union bpf_attr *attr,
|
||||
unsigned int size)
|
||||
{
|
||||
return syscall(__NR_bpf, cmd, attr, size);
|
||||
}
|
||||
|
||||
static inline int skel_closenz(int fd)
|
||||
{
|
||||
if (fd > 0)
|
||||
return close(fd);
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
static inline int bpf_load_and_run(struct bpf_load_and_run_opts *opts)
|
||||
{
|
||||
int map_fd = -1, prog_fd = -1, key = 0, err;
|
||||
union bpf_attr attr;
|
||||
|
||||
map_fd = bpf_create_map_name(BPF_MAP_TYPE_ARRAY, "__loader.map", 4,
|
||||
opts->data_sz, 1, 0);
|
||||
if (map_fd < 0) {
|
||||
opts->errstr = "failed to create loader map";
|
||||
err = -errno;
|
||||
goto out;
|
||||
}
|
||||
|
||||
err = bpf_map_update_elem(map_fd, &key, opts->data, 0);
|
||||
if (err < 0) {
|
||||
opts->errstr = "failed to update loader map";
|
||||
err = -errno;
|
||||
goto out;
|
||||
}
|
||||
|
||||
memset(&attr, 0, sizeof(attr));
|
||||
attr.prog_type = BPF_PROG_TYPE_SYSCALL;
|
||||
attr.insns = (long) opts->insns;
|
||||
attr.insn_cnt = opts->insns_sz / sizeof(struct bpf_insn);
|
||||
attr.license = (long) "Dual BSD/GPL";
|
||||
memcpy(attr.prog_name, "__loader.prog", sizeof("__loader.prog"));
|
||||
attr.fd_array = (long) &map_fd;
|
||||
attr.log_level = opts->ctx->log_level;
|
||||
attr.log_size = opts->ctx->log_size;
|
||||
attr.log_buf = opts->ctx->log_buf;
|
||||
attr.prog_flags = BPF_F_SLEEPABLE;
|
||||
prog_fd = skel_sys_bpf(BPF_PROG_LOAD, &attr, sizeof(attr));
|
||||
if (prog_fd < 0) {
|
||||
opts->errstr = "failed to load loader prog";
|
||||
err = -errno;
|
||||
goto out;
|
||||
}
|
||||
|
||||
memset(&attr, 0, sizeof(attr));
|
||||
attr.test.prog_fd = prog_fd;
|
||||
attr.test.ctx_in = (long) opts->ctx;
|
||||
attr.test.ctx_size_in = opts->ctx->sz;
|
||||
err = skel_sys_bpf(BPF_PROG_TEST_RUN, &attr, sizeof(attr));
|
||||
if (err < 0 || (int)attr.test.retval < 0) {
|
||||
opts->errstr = "failed to execute loader prog";
|
||||
if (err < 0)
|
||||
err = -errno;
|
||||
else
|
||||
err = (int)attr.test.retval;
|
||||
goto out;
|
||||
}
|
||||
err = 0;
|
||||
out:
|
||||
if (map_fd >= 0)
|
||||
close(map_fd);
|
||||
if (prog_fd >= 0)
|
||||
close(prog_fd);
|
||||
return err;
|
||||
}
|
||||
|
||||
#endif
|
||||
1
tools/testing/selftests/bpf/.gitignore
vendored
1
tools/testing/selftests/bpf/.gitignore
vendored
@ -30,6 +30,7 @@ test_sysctl
|
||||
xdping
|
||||
test_cpp
|
||||
*.skel.h
|
||||
*.lskel.h
|
||||
/no_alu32
|
||||
/bpf_gcc
|
||||
/tools
|
||||
|
||||
@ -312,6 +312,10 @@ SKEL_BLACKLIST := btf__% test_pinning_invalid.c test_sk_assign.c
|
||||
LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \
|
||||
linked_vars.skel.h linked_maps.skel.h
|
||||
|
||||
LSKELS := kfunc_call_test.c fentry_test.c fexit_test.c fexit_sleep.c \
|
||||
test_ksyms_module.c test_ringbuf.c atomics.c trace_printk.c
|
||||
SKEL_BLACKLIST += $$(LSKELS)
|
||||
|
||||
test_static_linked.skel.h-deps := test_static_linked1.o test_static_linked2.o
|
||||
linked_funcs.skel.h-deps := linked_funcs1.o linked_funcs2.o
|
||||
linked_vars.skel.h-deps := linked_vars1.o linked_vars2.o
|
||||
@ -339,6 +343,7 @@ TRUNNER_BPF_OBJS := $$(patsubst %.c,$$(TRUNNER_OUTPUT)/%.o, $$(TRUNNER_BPF_SRCS)
|
||||
TRUNNER_BPF_SKELS := $$(patsubst %.c,$$(TRUNNER_OUTPUT)/%.skel.h, \
|
||||
$$(filter-out $(SKEL_BLACKLIST) $(LINKED_BPF_SRCS),\
|
||||
$$(TRUNNER_BPF_SRCS)))
|
||||
TRUNNER_BPF_LSKELS := $$(patsubst %.c,$$(TRUNNER_OUTPUT)/%.lskel.h, $$(LSKELS))
|
||||
TRUNNER_BPF_SKELS_LINKED := $$(addprefix $$(TRUNNER_OUTPUT)/,$(LINKED_SKELS))
|
||||
TEST_GEN_FILES += $$(TRUNNER_BPF_OBJS)
|
||||
|
||||
@ -380,6 +385,14 @@ $(TRUNNER_BPF_SKELS): %.skel.h: %.o $(BPFTOOL) | $(TRUNNER_OUTPUT)
|
||||
$(Q)diff $$(<:.o=.linked2.o) $$(<:.o=.linked3.o)
|
||||
$(Q)$$(BPFTOOL) gen skeleton $$(<:.o=.linked3.o) name $$(notdir $$(<:.o=)) > $$@
|
||||
|
||||
$(TRUNNER_BPF_LSKELS): %.lskel.h: %.o $(BPFTOOL) | $(TRUNNER_OUTPUT)
|
||||
$$(call msg,GEN-SKEL,$(TRUNNER_BINARY),$$@)
|
||||
$(Q)$$(BPFTOOL) gen object $$(<:.o=.linked1.o) $$<
|
||||
$(Q)$$(BPFTOOL) gen object $$(<:.o=.linked2.o) $$(<:.o=.linked1.o)
|
||||
$(Q)$$(BPFTOOL) gen object $$(<:.o=.linked3.o) $$(<:.o=.linked2.o)
|
||||
$(Q)diff $$(<:.o=.linked2.o) $$(<:.o=.linked3.o)
|
||||
$(Q)$$(BPFTOOL) gen skeleton -L $$(<:.o=.linked3.o) name $$(notdir $$(<:.o=)) > $$@
|
||||
|
||||
$(TRUNNER_BPF_SKELS_LINKED): $(TRUNNER_BPF_OBJS) $(BPFTOOL) | $(TRUNNER_OUTPUT)
|
||||
$$(call msg,LINK-BPF,$(TRUNNER_BINARY),$$(@:.skel.h=.o))
|
||||
$(Q)$$(BPFTOOL) gen object $$(@:.skel.h=.linked1.o) $$(addprefix $(TRUNNER_OUTPUT)/,$$($$(@F)-deps))
|
||||
@ -409,6 +422,7 @@ $(TRUNNER_TEST_OBJS): $(TRUNNER_OUTPUT)/%.test.o: \
|
||||
$(TRUNNER_EXTRA_HDRS) \
|
||||
$(TRUNNER_BPF_OBJS) \
|
||||
$(TRUNNER_BPF_SKELS) \
|
||||
$(TRUNNER_BPF_LSKELS) \
|
||||
$(TRUNNER_BPF_SKELS_LINKED) \
|
||||
$$(BPFOBJ) | $(TRUNNER_OUTPUT)
|
||||
$$(call msg,TEST-OBJ,$(TRUNNER_BINARY),$$@)
|
||||
@ -516,6 +530,6 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o $(OUTPUT)/testing_helpers.o \
|
||||
EXTRA_CLEAN := $(TEST_CUSTOM_PROGS) $(SCRATCH_DIR) $(HOST_SCRATCH_DIR) \
|
||||
prog_tests/tests.h map_tests/tests.h verifier/tests.h \
|
||||
feature \
|
||||
$(addprefix $(OUTPUT)/,*.o *.skel.h no_alu32 bpf_gcc bpf_testmod.ko)
|
||||
$(addprefix $(OUTPUT)/,*.o *.skel.h *.lskel.h no_alu32 bpf_gcc bpf_testmod.ko)
|
||||
|
||||
.PHONY: docs docs-clean
|
||||
|
||||
@ -2,19 +2,19 @@
|
||||
|
||||
#include <test_progs.h>
|
||||
|
||||
#include "atomics.skel.h"
|
||||
#include "atomics.lskel.h"
|
||||
|
||||
static void test_add(struct atomics *skel)
|
||||
{
|
||||
int err, prog_fd;
|
||||
__u32 duration = 0, retval;
|
||||
struct bpf_link *link;
|
||||
int link_fd;
|
||||
|
||||
link = bpf_program__attach(skel->progs.add);
|
||||
if (CHECK(IS_ERR(link), "attach(add)", "err: %ld\n", PTR_ERR(link)))
|
||||
link_fd = atomics__add__attach(skel);
|
||||
if (!ASSERT_GT(link_fd, 0, "attach(add)"))
|
||||
return;
|
||||
|
||||
prog_fd = bpf_program__fd(skel->progs.add);
|
||||
prog_fd = skel->progs.add.prog_fd;
|
||||
err = bpf_prog_test_run(prog_fd, 1, NULL, 0,
|
||||
NULL, NULL, &retval, &duration);
|
||||
if (CHECK(err || retval, "test_run add",
|
||||
@ -33,20 +33,20 @@ static void test_add(struct atomics *skel)
|
||||
ASSERT_EQ(skel->data->add_noreturn_value, 3, "add_noreturn_value");
|
||||
|
||||
cleanup:
|
||||
bpf_link__destroy(link);
|
||||
close(link_fd);
|
||||
}
|
||||
|
||||
static void test_sub(struct atomics *skel)
|
||||
{
|
||||
int err, prog_fd;
|
||||
__u32 duration = 0, retval;
|
||||
struct bpf_link *link;
|
||||
int link_fd;
|
||||
|
||||
link = bpf_program__attach(skel->progs.sub);
|
||||
if (CHECK(IS_ERR(link), "attach(sub)", "err: %ld\n", PTR_ERR(link)))
|
||||
link_fd = atomics__sub__attach(skel);
|
||||
if (!ASSERT_GT(link_fd, 0, "attach(sub)"))
|
||||
return;
|
||||
|
||||
prog_fd = bpf_program__fd(skel->progs.sub);
|
||||
prog_fd = skel->progs.sub.prog_fd;
|
||||
err = bpf_prog_test_run(prog_fd, 1, NULL, 0,
|
||||
NULL, NULL, &retval, &duration);
|
||||
if (CHECK(err || retval, "test_run sub",
|
||||
@ -66,20 +66,20 @@ static void test_sub(struct atomics *skel)
|
||||
ASSERT_EQ(skel->data->sub_noreturn_value, -1, "sub_noreturn_value");
|
||||
|
||||
cleanup:
|
||||
bpf_link__destroy(link);
|
||||
close(link_fd);
|
||||
}
|
||||
|
||||
static void test_and(struct atomics *skel)
|
||||
{
|
||||
int err, prog_fd;
|
||||
__u32 duration = 0, retval;
|
||||
struct bpf_link *link;
|
||||
int link_fd;
|
||||
|
||||
link = bpf_program__attach(skel->progs.and);
|
||||
if (CHECK(IS_ERR(link), "attach(and)", "err: %ld\n", PTR_ERR(link)))
|
||||
link_fd = atomics__and__attach(skel);
|
||||
if (!ASSERT_GT(link_fd, 0, "attach(and)"))
|
||||
return;
|
||||
|
||||
prog_fd = bpf_program__fd(skel->progs.and);
|
||||
prog_fd = skel->progs.and.prog_fd;
|
||||
err = bpf_prog_test_run(prog_fd, 1, NULL, 0,
|
||||
NULL, NULL, &retval, &duration);
|
||||
if (CHECK(err || retval, "test_run and",
|
||||
@ -94,20 +94,20 @@ static void test_and(struct atomics *skel)
|
||||
|
||||
ASSERT_EQ(skel->data->and_noreturn_value, 0x010ull << 32, "and_noreturn_value");
|
||||
cleanup:
|
||||
bpf_link__destroy(link);
|
||||
close(link_fd);
|
||||
}
|
||||
|
||||
static void test_or(struct atomics *skel)
|
||||
{
|
||||
int err, prog_fd;
|
||||
__u32 duration = 0, retval;
|
||||
struct bpf_link *link;
|
||||
int link_fd;
|
||||
|
||||
link = bpf_program__attach(skel->progs.or);
|
||||
if (CHECK(IS_ERR(link), "attach(or)", "err: %ld\n", PTR_ERR(link)))
|
||||
link_fd = atomics__or__attach(skel);
|
||||
if (!ASSERT_GT(link_fd, 0, "attach(or)"))
|
||||
return;
|
||||
|
||||
prog_fd = bpf_program__fd(skel->progs.or);
|
||||
prog_fd = skel->progs.or.prog_fd;
|
||||
err = bpf_prog_test_run(prog_fd, 1, NULL, 0,
|
||||
NULL, NULL, &retval, &duration);
|
||||
if (CHECK(err || retval, "test_run or",
|
||||
@ -123,20 +123,20 @@ static void test_or(struct atomics *skel)
|
||||
|
||||
ASSERT_EQ(skel->data->or_noreturn_value, 0x111ull << 32, "or_noreturn_value");
|
||||
cleanup:
|
||||
bpf_link__destroy(link);
|
||||
close(link_fd);
|
||||
}
|
||||
|
||||
static void test_xor(struct atomics *skel)
|
||||
{
|
||||
int err, prog_fd;
|
||||
__u32 duration = 0, retval;
|
||||
struct bpf_link *link;
|
||||
int link_fd;
|
||||
|
||||
link = bpf_program__attach(skel->progs.xor);
|
||||
if (CHECK(IS_ERR(link), "attach(xor)", "err: %ld\n", PTR_ERR(link)))
|
||||
link_fd = atomics__xor__attach(skel);
|
||||
if (!ASSERT_GT(link_fd, 0, "attach(xor)"))
|
||||
return;
|
||||
|
||||
prog_fd = bpf_program__fd(skel->progs.xor);
|
||||
prog_fd = skel->progs.xor.prog_fd;
|
||||
err = bpf_prog_test_run(prog_fd, 1, NULL, 0,
|
||||
NULL, NULL, &retval, &duration);
|
||||
if (CHECK(err || retval, "test_run xor",
|
||||
@ -151,20 +151,20 @@ static void test_xor(struct atomics *skel)
|
||||
|
||||
ASSERT_EQ(skel->data->xor_noreturn_value, 0x101ull << 32, "xor_nxoreturn_value");
|
||||
cleanup:
|
||||
bpf_link__destroy(link);
|
||||
close(link_fd);
|
||||
}
|
||||
|
||||
static void test_cmpxchg(struct atomics *skel)
|
||||
{
|
||||
int err, prog_fd;
|
||||
__u32 duration = 0, retval;
|
||||
struct bpf_link *link;
|
||||
int link_fd;
|
||||
|
||||
link = bpf_program__attach(skel->progs.cmpxchg);
|
||||
if (CHECK(IS_ERR(link), "attach(cmpxchg)", "err: %ld\n", PTR_ERR(link)))
|
||||
link_fd = atomics__cmpxchg__attach(skel);
|
||||
if (!ASSERT_GT(link_fd, 0, "attach(cmpxchg)"))
|
||||
return;
|
||||
|
||||
prog_fd = bpf_program__fd(skel->progs.cmpxchg);
|
||||
prog_fd = skel->progs.cmpxchg.prog_fd;
|
||||
err = bpf_prog_test_run(prog_fd, 1, NULL, 0,
|
||||
NULL, NULL, &retval, &duration);
|
||||
if (CHECK(err || retval, "test_run add",
|
||||
@ -180,20 +180,20 @@ static void test_cmpxchg(struct atomics *skel)
|
||||
ASSERT_EQ(skel->bss->cmpxchg32_result_succeed, 1, "cmpxchg_result_succeed");
|
||||
|
||||
cleanup:
|
||||
bpf_link__destroy(link);
|
||||
close(link_fd);
|
||||
}
|
||||
|
||||
static void test_xchg(struct atomics *skel)
|
||||
{
|
||||
int err, prog_fd;
|
||||
__u32 duration = 0, retval;
|
||||
struct bpf_link *link;
|
||||
int link_fd;
|
||||
|
||||
link = bpf_program__attach(skel->progs.xchg);
|
||||
if (CHECK(IS_ERR(link), "attach(xchg)", "err: %ld\n", PTR_ERR(link)))
|
||||
link_fd = atomics__xchg__attach(skel);
|
||||
if (!ASSERT_GT(link_fd, 0, "attach(xchg)"))
|
||||
return;
|
||||
|
||||
prog_fd = bpf_program__fd(skel->progs.xchg);
|
||||
prog_fd = skel->progs.xchg.prog_fd;
|
||||
err = bpf_prog_test_run(prog_fd, 1, NULL, 0,
|
||||
NULL, NULL, &retval, &duration);
|
||||
if (CHECK(err || retval, "test_run add",
|
||||
@ -207,7 +207,7 @@ static void test_xchg(struct atomics *skel)
|
||||
ASSERT_EQ(skel->bss->xchg32_result, 1, "xchg32_result");
|
||||
|
||||
cleanup:
|
||||
bpf_link__destroy(link);
|
||||
close(link_fd);
|
||||
}
|
||||
|
||||
void test_atomics(void)
|
||||
|
||||
@ -1,8 +1,8 @@
|
||||
// SPDX-License-Identifier: GPL-2.0
|
||||
/* Copyright (c) 2019 Facebook */
|
||||
#include <test_progs.h>
|
||||
#include "fentry_test.skel.h"
|
||||
#include "fexit_test.skel.h"
|
||||
#include "fentry_test.lskel.h"
|
||||
#include "fexit_test.lskel.h"
|
||||
|
||||
void test_fentry_fexit(void)
|
||||
{
|
||||
@ -26,7 +26,7 @@ void test_fentry_fexit(void)
|
||||
if (CHECK(err, "fexit_attach", "fexit attach failed: %d\n", err))
|
||||
goto close_prog;
|
||||
|
||||
prog_fd = bpf_program__fd(fexit_skel->progs.test1);
|
||||
prog_fd = fexit_skel->progs.test1.prog_fd;
|
||||
err = bpf_prog_test_run(prog_fd, 1, NULL, 0,
|
||||
NULL, NULL, &retval, &duration);
|
||||
CHECK(err || retval, "ipv6",
|
||||
|
||||
@ -1,13 +1,13 @@
|
||||
// SPDX-License-Identifier: GPL-2.0
|
||||
/* Copyright (c) 2019 Facebook */
|
||||
#include <test_progs.h>
|
||||
#include "fentry_test.skel.h"
|
||||
#include "fentry_test.lskel.h"
|
||||
|
||||
static int fentry_test(struct fentry_test *fentry_skel)
|
||||
{
|
||||
int err, prog_fd, i;
|
||||
__u32 duration = 0, retval;
|
||||
struct bpf_link *link;
|
||||
int link_fd;
|
||||
__u64 *result;
|
||||
|
||||
err = fentry_test__attach(fentry_skel);
|
||||
@ -15,11 +15,11 @@ static int fentry_test(struct fentry_test *fentry_skel)
|
||||
return err;
|
||||
|
||||
/* Check that already linked program can't be attached again. */
|
||||
link = bpf_program__attach(fentry_skel->progs.test1);
|
||||
if (!ASSERT_ERR_PTR(link, "fentry_attach_link"))
|
||||
link_fd = fentry_test__test1__attach(fentry_skel);
|
||||
if (!ASSERT_LT(link_fd, 0, "fentry_attach_link"))
|
||||
return -1;
|
||||
|
||||
prog_fd = bpf_program__fd(fentry_skel->progs.test1);
|
||||
prog_fd = fentry_skel->progs.test1.prog_fd;
|
||||
err = bpf_prog_test_run(prog_fd, 1, NULL, 0,
|
||||
NULL, NULL, &retval, &duration);
|
||||
ASSERT_OK(err, "test_run");
|
||||
|
||||
@ -6,7 +6,7 @@
|
||||
#include <time.h>
|
||||
#include <sys/mman.h>
|
||||
#include <sys/syscall.h>
|
||||
#include "fexit_sleep.skel.h"
|
||||
#include "fexit_sleep.lskel.h"
|
||||
|
||||
static int do_sleep(void *skel)
|
||||
{
|
||||
@ -58,8 +58,8 @@ void test_fexit_sleep(void)
|
||||
* waiting for percpu_ref_kill to confirm). The other one
|
||||
* will be freed quickly.
|
||||
*/
|
||||
close(bpf_program__fd(fexit_skel->progs.nanosleep_fentry));
|
||||
close(bpf_program__fd(fexit_skel->progs.nanosleep_fexit));
|
||||
close(fexit_skel->progs.nanosleep_fentry.prog_fd);
|
||||
close(fexit_skel->progs.nanosleep_fexit.prog_fd);
|
||||
fexit_sleep__detach(fexit_skel);
|
||||
|
||||
/* kill the thread to unwind sys_nanosleep stack through the trampoline */
|
||||
|
||||
@ -1,13 +1,13 @@
|
||||
// SPDX-License-Identifier: GPL-2.0
|
||||
/* Copyright (c) 2019 Facebook */
|
||||
#include <test_progs.h>
|
||||
#include "fexit_test.skel.h"
|
||||
#include "fexit_test.lskel.h"
|
||||
|
||||
static int fexit_test(struct fexit_test *fexit_skel)
|
||||
{
|
||||
int err, prog_fd, i;
|
||||
__u32 duration = 0, retval;
|
||||
struct bpf_link *link;
|
||||
int link_fd;
|
||||
__u64 *result;
|
||||
|
||||
err = fexit_test__attach(fexit_skel);
|
||||
@ -15,11 +15,11 @@ static int fexit_test(struct fexit_test *fexit_skel)
|
||||
return err;
|
||||
|
||||
/* Check that already linked program can't be attached again. */
|
||||
link = bpf_program__attach(fexit_skel->progs.test1);
|
||||
if (!ASSERT_ERR_PTR(link, "fexit_attach_link"))
|
||||
link_fd = fexit_test__test1__attach(fexit_skel);
|
||||
if (!ASSERT_LT(link_fd, 0, "fexit_attach_link"))
|
||||
return -1;
|
||||
|
||||
prog_fd = bpf_program__fd(fexit_skel->progs.test1);
|
||||
prog_fd = fexit_skel->progs.test1.prog_fd;
|
||||
err = bpf_prog_test_run(prog_fd, 1, NULL, 0,
|
||||
NULL, NULL, &retval, &duration);
|
||||
ASSERT_OK(err, "test_run");
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
/* Copyright (c) 2021 Facebook */
|
||||
#include <test_progs.h>
|
||||
#include <network_helpers.h>
|
||||
#include "kfunc_call_test.skel.h"
|
||||
#include "kfunc_call_test.lskel.h"
|
||||
#include "kfunc_call_test_subprog.skel.h"
|
||||
|
||||
static void test_main(void)
|
||||
@ -14,13 +14,13 @@ static void test_main(void)
|
||||
if (!ASSERT_OK_PTR(skel, "skel"))
|
||||
return;
|
||||
|
||||
prog_fd = bpf_program__fd(skel->progs.kfunc_call_test1);
|
||||
prog_fd = skel->progs.kfunc_call_test1.prog_fd;
|
||||
err = bpf_prog_test_run(prog_fd, 1, &pkt_v4, sizeof(pkt_v4),
|
||||
NULL, NULL, (__u32 *)&retval, NULL);
|
||||
ASSERT_OK(err, "bpf_prog_test_run(test1)");
|
||||
ASSERT_EQ(retval, 12, "test1-retval");
|
||||
|
||||
prog_fd = bpf_program__fd(skel->progs.kfunc_call_test2);
|
||||
prog_fd = skel->progs.kfunc_call_test2.prog_fd;
|
||||
err = bpf_prog_test_run(prog_fd, 1, &pkt_v4, sizeof(pkt_v4),
|
||||
NULL, NULL, (__u32 *)&retval, NULL);
|
||||
ASSERT_OK(err, "bpf_prog_test_run(test2)");
|
||||
|
||||
@ -4,7 +4,7 @@
|
||||
#include <test_progs.h>
|
||||
#include <bpf/libbpf.h>
|
||||
#include <bpf/btf.h>
|
||||
#include "test_ksyms_module.skel.h"
|
||||
#include "test_ksyms_module.lskel.h"
|
||||
|
||||
static int duration;
|
||||
|
||||
|
||||
@ -12,7 +12,7 @@
|
||||
#include <sys/sysinfo.h>
|
||||
#include <linux/perf_event.h>
|
||||
#include <linux/ring_buffer.h>
|
||||
#include "test_ringbuf.skel.h"
|
||||
#include "test_ringbuf.lskel.h"
|
||||
|
||||
#define EDONE 7777
|
||||
|
||||
@ -93,9 +93,7 @@ void test_ringbuf(void)
|
||||
if (CHECK(!skel, "skel_open", "skeleton open failed\n"))
|
||||
return;
|
||||
|
||||
err = bpf_map__set_max_entries(skel->maps.ringbuf, page_size);
|
||||
if (CHECK(err != 0, "bpf_map__set_max_entries", "bpf_map__set_max_entries failed\n"))
|
||||
goto cleanup;
|
||||
skel->maps.ringbuf.max_entries = page_size;
|
||||
|
||||
err = test_ringbuf__load(skel);
|
||||
if (CHECK(err != 0, "skel_load", "skeleton load failed\n"))
|
||||
@ -104,7 +102,7 @@ void test_ringbuf(void)
|
||||
/* only trigger BPF program for current process */
|
||||
skel->bss->pid = getpid();
|
||||
|
||||
ringbuf = ring_buffer__new(bpf_map__fd(skel->maps.ringbuf),
|
||||
ringbuf = ring_buffer__new(skel->maps.ringbuf.map_fd,
|
||||
process_sample, NULL, NULL);
|
||||
if (CHECK(!ringbuf, "ringbuf_create", "failed to create ringbuf\n"))
|
||||
goto cleanup;
|
||||
|
||||
55
tools/testing/selftests/bpf/prog_tests/syscall.c
Normal file
55
tools/testing/selftests/bpf/prog_tests/syscall.c
Normal file
@ -0,0 +1,55 @@
|
||||
// SPDX-License-Identifier: GPL-2.0
|
||||
/* Copyright (c) 2021 Facebook */
|
||||
#include <test_progs.h>
|
||||
#include "syscall.skel.h"
|
||||
|
||||
struct args {
|
||||
__u64 log_buf;
|
||||
__u32 log_size;
|
||||
int max_entries;
|
||||
int map_fd;
|
||||
int prog_fd;
|
||||
int btf_fd;
|
||||
};
|
||||
|
||||
void test_syscall(void)
|
||||
{
|
||||
static char verifier_log[8192];
|
||||
struct args ctx = {
|
||||
.max_entries = 1024,
|
||||
.log_buf = (uintptr_t) verifier_log,
|
||||
.log_size = sizeof(verifier_log),
|
||||
};
|
||||
struct bpf_prog_test_run_attr tattr = {
|
||||
.ctx_in = &ctx,
|
||||
.ctx_size_in = sizeof(ctx),
|
||||
};
|
||||
struct syscall *skel = NULL;
|
||||
__u64 key = 12, value = 0;
|
||||
int err;
|
||||
|
||||
skel = syscall__open_and_load();
|
||||
if (!ASSERT_OK_PTR(skel, "skel_load"))
|
||||
goto cleanup;
|
||||
|
||||
tattr.prog_fd = bpf_program__fd(skel->progs.bpf_prog);
|
||||
err = bpf_prog_test_run_xattr(&tattr);
|
||||
ASSERT_EQ(err, 0, "err");
|
||||
ASSERT_EQ(tattr.retval, 1, "retval");
|
||||
ASSERT_GT(ctx.map_fd, 0, "ctx.map_fd");
|
||||
ASSERT_GT(ctx.prog_fd, 0, "ctx.prog_fd");
|
||||
ASSERT_OK(memcmp(verifier_log, "processed", sizeof("processed") - 1),
|
||||
"verifier_log");
|
||||
|
||||
err = bpf_map_lookup_elem(ctx.map_fd, &key, &value);
|
||||
ASSERT_EQ(err, 0, "map_lookup");
|
||||
ASSERT_EQ(value, 34, "map lookup value");
|
||||
cleanup:
|
||||
syscall__destroy(skel);
|
||||
if (ctx.prog_fd > 0)
|
||||
close(ctx.prog_fd);
|
||||
if (ctx.map_fd > 0)
|
||||
close(ctx.map_fd);
|
||||
if (ctx.btf_fd > 0)
|
||||
close(ctx.btf_fd);
|
||||
}
|
||||
@ -3,7 +3,7 @@
|
||||
|
||||
#include <test_progs.h>
|
||||
|
||||
#include "trace_printk.skel.h"
|
||||
#include "trace_printk.lskel.h"
|
||||
|
||||
#define TRACEBUF "/sys/kernel/debug/tracing/trace_pipe"
|
||||
#define SEARCHMSG "testing,testing"
|
||||
@ -21,6 +21,9 @@ void test_trace_printk(void)
|
||||
if (CHECK(!skel, "skel_open", "failed to open skeleton\n"))
|
||||
return;
|
||||
|
||||
ASSERT_EQ(skel->rodata->fmt[0], 'T', "invalid printk fmt string");
|
||||
skel->rodata->fmt[0] = 't';
|
||||
|
||||
err = trace_printk__load(skel);
|
||||
if (CHECK(err, "skel_load", "failed to load skeleton: %d\n", err))
|
||||
goto cleanup;
|
||||
|
||||
121
tools/testing/selftests/bpf/progs/syscall.c
Normal file
121
tools/testing/selftests/bpf/progs/syscall.c
Normal file
@ -0,0 +1,121 @@
|
||||
// SPDX-License-Identifier: GPL-2.0
|
||||
/* Copyright (c) 2021 Facebook */
|
||||
#include <linux/stddef.h>
|
||||
#include <linux/bpf.h>
|
||||
#include <bpf/bpf_helpers.h>
|
||||
#include <bpf/bpf_tracing.h>
|
||||
#include <../../../tools/include/linux/filter.h>
|
||||
#include <linux/btf.h>
|
||||
|
||||
char _license[] SEC("license") = "GPL";
|
||||
|
||||
struct args {
|
||||
__u64 log_buf;
|
||||
__u32 log_size;
|
||||
int max_entries;
|
||||
int map_fd;
|
||||
int prog_fd;
|
||||
int btf_fd;
|
||||
};
|
||||
|
||||
#define BTF_INFO_ENC(kind, kind_flag, vlen) \
|
||||
((!!(kind_flag) << 31) | ((kind) << 24) | ((vlen) & BTF_MAX_VLEN))
|
||||
#define BTF_TYPE_ENC(name, info, size_or_type) (name), (info), (size_or_type)
|
||||
#define BTF_INT_ENC(encoding, bits_offset, nr_bits) \
|
||||
((encoding) << 24 | (bits_offset) << 16 | (nr_bits))
|
||||
#define BTF_TYPE_INT_ENC(name, encoding, bits_offset, bits, sz) \
|
||||
BTF_TYPE_ENC(name, BTF_INFO_ENC(BTF_KIND_INT, 0, 0), sz), \
|
||||
BTF_INT_ENC(encoding, bits_offset, bits)
|
||||
|
||||
static int btf_load(void)
|
||||
{
|
||||
struct btf_blob {
|
||||
struct btf_header btf_hdr;
|
||||
__u32 types[8];
|
||||
__u32 str;
|
||||
} raw_btf = {
|
||||
.btf_hdr = {
|
||||
.magic = BTF_MAGIC,
|
||||
.version = BTF_VERSION,
|
||||
.hdr_len = sizeof(struct btf_header),
|
||||
.type_len = sizeof(__u32) * 8,
|
||||
.str_off = sizeof(__u32) * 8,
|
||||
.str_len = sizeof(__u32),
|
||||
},
|
||||
.types = {
|
||||
/* long */
|
||||
BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 64, 8), /* [1] */
|
||||
/* unsigned long */
|
||||
BTF_TYPE_INT_ENC(0, 0, 0, 64, 8), /* [2] */
|
||||
},
|
||||
};
|
||||
static union bpf_attr btf_load_attr = {
|
||||
.btf_size = sizeof(raw_btf),
|
||||
};
|
||||
|
||||
btf_load_attr.btf = (long)&raw_btf;
|
||||
return bpf_sys_bpf(BPF_BTF_LOAD, &btf_load_attr, sizeof(btf_load_attr));
|
||||
}
|
||||
|
||||
SEC("syscall")
|
||||
int bpf_prog(struct args *ctx)
|
||||
{
|
||||
static char license[] = "GPL";
|
||||
static struct bpf_insn insns[] = {
|
||||
BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0),
|
||||
BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
|
||||
BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
|
||||
BPF_LD_MAP_FD(BPF_REG_1, 0),
|
||||
BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
|
||||
BPF_MOV64_IMM(BPF_REG_0, 0),
|
||||
BPF_EXIT_INSN(),
|
||||
};
|
||||
static union bpf_attr map_create_attr = {
|
||||
.map_type = BPF_MAP_TYPE_HASH,
|
||||
.key_size = 8,
|
||||
.value_size = 8,
|
||||
.btf_key_type_id = 1,
|
||||
.btf_value_type_id = 2,
|
||||
};
|
||||
static union bpf_attr map_update_attr = { .map_fd = 1, };
|
||||
static __u64 key = 12;
|
||||
static __u64 value = 34;
|
||||
static union bpf_attr prog_load_attr = {
|
||||
.prog_type = BPF_PROG_TYPE_XDP,
|
||||
.insn_cnt = sizeof(insns) / sizeof(insns[0]),
|
||||
};
|
||||
int ret;
|
||||
|
||||
ret = btf_load();
|
||||
if (ret <= 0)
|
||||
return ret;
|
||||
|
||||
ctx->btf_fd = ret;
|
||||
map_create_attr.max_entries = ctx->max_entries;
|
||||
map_create_attr.btf_fd = ret;
|
||||
|
||||
prog_load_attr.license = (long) license;
|
||||
prog_load_attr.insns = (long) insns;
|
||||
prog_load_attr.log_buf = ctx->log_buf;
|
||||
prog_load_attr.log_size = ctx->log_size;
|
||||
prog_load_attr.log_level = 1;
|
||||
|
||||
ret = bpf_sys_bpf(BPF_MAP_CREATE, &map_create_attr, sizeof(map_create_attr));
|
||||
if (ret <= 0)
|
||||
return ret;
|
||||
ctx->map_fd = ret;
|
||||
insns[3].imm = ret;
|
||||
|
||||
map_update_attr.map_fd = ret;
|
||||
map_update_attr.key = (long) &key;
|
||||
map_update_attr.value = (long) &value;
|
||||
ret = bpf_sys_bpf(BPF_MAP_UPDATE_ELEM, &map_update_attr, sizeof(map_update_attr));
|
||||
if (ret < 0)
|
||||
return ret;
|
||||
|
||||
ret = bpf_sys_bpf(BPF_PROG_LOAD, &prog_load_attr, sizeof(prog_load_attr));
|
||||
if (ret <= 0)
|
||||
return ret;
|
||||
ctx->prog_fd = ret;
|
||||
return 1;
|
||||
}
|
||||
@ -35,7 +35,7 @@ long prod_pos = 0;
|
||||
/* inner state */
|
||||
long seq = 0;
|
||||
|
||||
SEC("tp/syscalls/sys_enter_getpgid")
|
||||
SEC("fentry/__x64_sys_getpgid")
|
||||
int test_ringbuf(void *ctx)
|
||||
{
|
||||
int cur_pid = bpf_get_current_pid_tgid() >> 32;
|
||||
@ -48,7 +48,7 @@ int test_ringbuf(void *ctx)
|
||||
sample = bpf_ringbuf_reserve(&ringbuf, sizeof(*sample), 0);
|
||||
if (!sample) {
|
||||
__sync_fetch_and_add(&dropped, 1);
|
||||
return 1;
|
||||
return 0;
|
||||
}
|
||||
|
||||
sample->pid = pid;
|
||||
|
||||
@ -4,8 +4,18 @@
|
||||
|
||||
const char LICENSE[] SEC("license") = "GPL";
|
||||
|
||||
struct {
|
||||
__uint(type, BPF_MAP_TYPE_ARRAY);
|
||||
__uint(max_entries, 1);
|
||||
__type(key, __u32);
|
||||
__type(value, __u64);
|
||||
} array SEC(".maps");
|
||||
|
||||
__noinline int sub1(int x)
|
||||
{
|
||||
int key = 0;
|
||||
|
||||
bpf_map_lookup_elem(&array, &key);
|
||||
return x + 1;
|
||||
}
|
||||
|
||||
@ -23,6 +33,9 @@ static __noinline int sub3(int z)
|
||||
|
||||
static __noinline int sub4(int w)
|
||||
{
|
||||
int key = 0;
|
||||
|
||||
bpf_map_lookup_elem(&array, &key);
|
||||
return w + sub3(5) + sub1(6);
|
||||
}
|
||||
|
||||
|
||||
@ -10,11 +10,11 @@ char _license[] SEC("license") = "GPL";
|
||||
int trace_printk_ret = 0;
|
||||
int trace_printk_ran = 0;
|
||||
|
||||
SEC("tp/raw_syscalls/sys_enter")
|
||||
const char fmt[] = "Testing,testing %d\n";
|
||||
|
||||
SEC("fentry/__x64_sys_nanosleep")
|
||||
int sys_enter(void *ctx)
|
||||
{
|
||||
static const char fmt[] = "testing,testing %d\n";
|
||||
|
||||
trace_printk_ret = bpf_trace_printk(fmt, sizeof(fmt),
|
||||
++trace_printk_ran);
|
||||
return 0;
|
||||
|
||||
Loading…
Reference in New Issue
Block a user