libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
/* CTF file creation.
|
2020-01-01 15:57:01 +08:00
|
|
|
Copyright (C) 2019-2020 Free Software Foundation, Inc.
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
|
|
|
This file is part of libctf.
|
|
|
|
|
|
|
|
libctf is free software; you can redistribute it and/or modify it under
|
|
|
|
the terms of the GNU General Public License as published by the Free
|
|
|
|
Software Foundation; either version 3, or (at your option) any later
|
|
|
|
version.
|
|
|
|
|
|
|
|
This program is distributed in the hope that it will be useful, but
|
|
|
|
WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
|
|
|
|
See the GNU General Public License for more details.
|
|
|
|
|
|
|
|
You should have received a copy of the GNU General Public License
|
|
|
|
along with this program; see the file COPYING. If not see
|
|
|
|
<http://www.gnu.org/licenses/>. */
|
|
|
|
|
|
|
|
#include <ctf-impl.h>
|
|
|
|
#include <sys/param.h>
|
|
|
|
#include <assert.h>
|
|
|
|
#include <string.h>
|
2019-12-16 19:38:57 +08:00
|
|
|
#include <unistd.h>
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
#include <zlib.h>
|
|
|
|
|
libctf: fix a number of build problems found on Solaris and NetBSD
- Use of nonportable <endian.h>
- Use of qsort_r
- Use of zlib without appropriate magic to pull in the binutils zlib
- Use of off64_t without checking (fixed by dropping the unused fields
that need off64_t entirely)
- signedness problems due to long being too short a type on 32-bit
platforms: ctf_id_t is now 'unsigned long', and CTF_ERR must be
used only for functions that return ctf_id_t
- One lingering use of bzero() and of <sys/errno.h>
All fixed, using code from gnulib where possible.
Relatedly, set cts_size in a couple of places it was missed
(string table and symbol table loading upon ctf_bfdopen()).
binutils/
* objdump.c (make_ctfsect): Drop cts_type, cts_flags, and
cts_offset.
* readelf.c (shdr_to_ctf_sect): Likewise.
include/
* ctf-api.h (ctf_sect_t): Drop cts_type, cts_flags, and cts_offset.
(ctf_id_t): This is now an unsigned type.
(CTF_ERR): Cast it to ctf_id_t. Note that it should only be used
for ctf_id_t-returning functions.
libctf/
* Makefile.am (ZLIB): New.
(ZLIBINC): Likewise.
(AM_CFLAGS): Use them.
(libctf_a_LIBADD): New, for LIBOBJS.
* configure.ac: Check for zlib, endian.h, and qsort_r.
* ctf-endian.h: New, providing htole64 and le64toh.
* swap.h: Code style fixes.
(bswap_identity_64): New.
* qsort_r.c: New, from gnulib (with one added #include).
* ctf-decls.h: New, providing a conditional qsort_r declaration,
and unconditional definitions of MIN and MAX.
* ctf-impl.h: Use it. Do not use <sys/errno.h>.
(ctf_set_errno): Now returns unsigned long.
* ctf-util.c (ctf_set_errno): Adjust here too.
* ctf-archive.c: Use ctf-endian.h.
(ctf_arc_open_by_offset): Use memset, not bzero. Drop cts_type,
cts_flags and cts_offset.
(ctf_arc_write): Drop debugging dependent on the size of off_t.
* ctf-create.c: Provide a definition of roundup if not defined.
(ctf_create): Drop cts_type, cts_flags and cts_offset.
(ctf_add_reftype): Do not check if type IDs are below zero.
(ctf_add_slice): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_member_offset): Cast error-returning ssize_t's to size_t
when known error-free. Drop CTF_ERR usage for functions returning
int.
(ctf_add_member_encoded): Drop CTF_ERR usage for functions returning
int.
(ctf_add_variable): Likewise.
(enumcmp): Likewise.
(enumadd): Likewise.
(membcmp): Likewise.
(ctf_add_type): Likewise. Cast error-returning ssize_t's to size_t
when known error-free.
* ctf-dump.c (ctf_is_slice): Drop CTF_ERR usage for functions
returning int: use CTF_ERR for functions returning ctf_type_id.
(ctf_dump_label): Likewise.
(ctf_dump_objts): Likewise.
* ctf-labels.c (ctf_label_topmost): Likewise.
(ctf_label_iter): Likewise.
(ctf_label_info): Likewise.
* ctf-lookup.c (ctf_func_args): Likewise.
* ctf-open.c (upgrade_types): Cast to size_t where appropriate.
(ctf_bufopen): Likewise. Use zlib types as needed.
* ctf-types.c (ctf_member_iter): Drop CTF_ERR usage for functions
returning int.
(ctf_enum_iter): Likewise.
(ctf_type_size): Likewise.
(ctf_type_align): Likewise. Cast to size_t where appropriate.
(ctf_type_kind_unsliced): Likewise.
(ctf_type_kind): Likewise.
(ctf_type_encoding): Likewise.
(ctf_member_info): Likewise.
(ctf_array_info): Likewise.
(ctf_enum_value): Likewise.
(ctf_type_rvisit): Likewise.
* ctf-open-bfd.c (ctf_bfdopen): Drop cts_type, cts_flags and
cts_offset.
(ctf_simple_open): Likewise.
(ctf_bfdopen_ctfsect): Likewise. Set cts_size properly.
* Makefile.in: Regenerate.
* aclocal.m4: Likewise.
* config.h: Likewise.
* configure: Likewise.
2019-05-31 17:10:51 +08:00
|
|
|
#ifndef roundup
|
|
|
|
#define roundup(x, y) ((((x) + ((y) - 1)) / (y)) * (y))
|
|
|
|
#endif
|
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
/* Make sure the ptrtab has enough space for at least one more type.
|
|
|
|
|
|
|
|
We start with 4KiB of ptrtab, enough for a thousand types, then grow it 25%
|
|
|
|
at a time. */
|
|
|
|
|
|
|
|
static int
|
|
|
|
ctf_grow_ptrtab (ctf_file_t *fp)
|
|
|
|
{
|
|
|
|
size_t new_ptrtab_len = fp->ctf_ptrtab_len;
|
|
|
|
|
|
|
|
/* We allocate one more ptrtab entry than we need, for the initial zero,
|
|
|
|
plus one because the caller will probably allocate a new type. */
|
|
|
|
|
|
|
|
if (fp->ctf_ptrtab == NULL)
|
|
|
|
new_ptrtab_len = 1024;
|
|
|
|
else if ((fp->ctf_typemax + 2) > fp->ctf_ptrtab_len)
|
|
|
|
new_ptrtab_len = fp->ctf_ptrtab_len * 1.25;
|
|
|
|
|
|
|
|
if (new_ptrtab_len != fp->ctf_ptrtab_len)
|
|
|
|
{
|
|
|
|
uint32_t *new_ptrtab;
|
|
|
|
|
|
|
|
if ((new_ptrtab = realloc (fp->ctf_ptrtab,
|
|
|
|
new_ptrtab_len * sizeof (uint32_t))) == NULL)
|
|
|
|
return (ctf_set_errno (fp, ENOMEM));
|
|
|
|
|
|
|
|
fp->ctf_ptrtab = new_ptrtab;
|
|
|
|
memset (fp->ctf_ptrtab + fp->ctf_ptrtab_len, 0,
|
|
|
|
(new_ptrtab_len - fp->ctf_ptrtab_len) * sizeof (uint32_t));
|
|
|
|
fp->ctf_ptrtab_len = new_ptrtab_len;
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
/* To create an empty CTF container, we just declare a zeroed header and call
|
|
|
|
ctf_bufopen() on it. If ctf_bufopen succeeds, we mark the new container r/w
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
and initialize the dynamic members. We start assigning type IDs at 1 because
|
|
|
|
type ID 0 is used as a sentinel and a not-found indicator. */
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
|
|
|
ctf_file_t *
|
|
|
|
ctf_create (int *errp)
|
|
|
|
{
|
|
|
|
static const ctf_header_t hdr = { .cth_preamble = { CTF_MAGIC, CTF_VERSION, 0 } };
|
|
|
|
|
|
|
|
ctf_dynhash_t *dthash;
|
|
|
|
ctf_dynhash_t *dvhash;
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
ctf_dynhash_t *structs = NULL, *unions = NULL, *enums = NULL, *names = NULL;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
ctf_sect_t cts;
|
|
|
|
ctf_file_t *fp;
|
|
|
|
|
|
|
|
libctf_init_debug();
|
|
|
|
dthash = ctf_dynhash_create (ctf_hash_integer, ctf_hash_eq_integer,
|
|
|
|
NULL, NULL);
|
|
|
|
if (dthash == NULL)
|
|
|
|
{
|
|
|
|
ctf_set_open_errno (errp, EAGAIN);
|
|
|
|
goto err;
|
|
|
|
}
|
|
|
|
|
|
|
|
dvhash = ctf_dynhash_create (ctf_hash_string, ctf_hash_eq_string,
|
|
|
|
NULL, NULL);
|
|
|
|
if (dvhash == NULL)
|
|
|
|
{
|
|
|
|
ctf_set_open_errno (errp, EAGAIN);
|
|
|
|
goto err_dt;
|
|
|
|
}
|
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
structs = ctf_dynhash_create (ctf_hash_string, ctf_hash_eq_string,
|
|
|
|
NULL, NULL);
|
|
|
|
unions = ctf_dynhash_create (ctf_hash_string, ctf_hash_eq_string,
|
|
|
|
NULL, NULL);
|
|
|
|
enums = ctf_dynhash_create (ctf_hash_string, ctf_hash_eq_string,
|
|
|
|
NULL, NULL);
|
|
|
|
names = ctf_dynhash_create (ctf_hash_string, ctf_hash_eq_string,
|
|
|
|
NULL, NULL);
|
|
|
|
if (!structs || !unions || !enums || !names)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
{
|
|
|
|
ctf_set_open_errno (errp, EAGAIN);
|
|
|
|
goto err_dv;
|
|
|
|
}
|
|
|
|
|
|
|
|
cts.cts_name = _CTF_SECTION;
|
|
|
|
cts.cts_data = &hdr;
|
|
|
|
cts.cts_size = sizeof (hdr);
|
|
|
|
cts.cts_entsize = 1;
|
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
if ((fp = ctf_bufopen_internal (&cts, NULL, NULL, NULL, 1, errp)) == NULL)
|
|
|
|
goto err_dv;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
fp->ctf_structs.ctn_writable = structs;
|
|
|
|
fp->ctf_unions.ctn_writable = unions;
|
|
|
|
fp->ctf_enums.ctn_writable = enums;
|
|
|
|
fp->ctf_names.ctn_writable = names;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
fp->ctf_dthash = dthash;
|
|
|
|
fp->ctf_dvhash = dvhash;
|
|
|
|
fp->ctf_dtoldid = 0;
|
2019-06-29 05:11:14 +08:00
|
|
|
fp->ctf_snapshots = 1;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
fp->ctf_snapshot_lu = 0;
|
2020-06-03 03:39:15 +08:00
|
|
|
fp->ctf_flags |= LCTF_DIRTY;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
ctf_set_ctl_hashes (fp);
|
|
|
|
ctf_setmodel (fp, CTF_MODEL_NATIVE);
|
|
|
|
if (ctf_grow_ptrtab (fp) < 0)
|
|
|
|
{
|
|
|
|
ctf_set_open_errno (errp, ctf_errno (fp));
|
|
|
|
ctf_file_close (fp);
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return fp;
|
|
|
|
|
|
|
|
err_dv:
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
ctf_dynhash_destroy (structs);
|
|
|
|
ctf_dynhash_destroy (unions);
|
|
|
|
ctf_dynhash_destroy (enums);
|
|
|
|
ctf_dynhash_destroy (names);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
ctf_dynhash_destroy (dvhash);
|
|
|
|
err_dt:
|
|
|
|
ctf_dynhash_destroy (dthash);
|
|
|
|
err:
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
static unsigned char *
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
ctf_copy_smembers (ctf_file_t *fp, ctf_dtdef_t *dtd, unsigned char *t)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
{
|
|
|
|
ctf_dmdef_t *dmd = ctf_list_next (&dtd->dtd_u.dtu_members);
|
|
|
|
ctf_member_t ctm;
|
|
|
|
|
|
|
|
for (; dmd != NULL; dmd = ctf_list_next (dmd))
|
|
|
|
{
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
ctf_member_t *copied;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
ctm.ctm_name = 0;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
ctm.ctm_type = (uint32_t) dmd->dmd_type;
|
|
|
|
ctm.ctm_offset = (uint32_t) dmd->dmd_offset;
|
|
|
|
|
|
|
|
memcpy (t, &ctm, sizeof (ctm));
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
copied = (ctf_member_t *) t;
|
|
|
|
if (dmd->dmd_name)
|
|
|
|
ctf_str_add_ref (fp, dmd->dmd_name, &copied->ctm_name);
|
|
|
|
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
t += sizeof (ctm);
|
|
|
|
}
|
|
|
|
|
|
|
|
return t;
|
|
|
|
}
|
|
|
|
|
|
|
|
static unsigned char *
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
ctf_copy_lmembers (ctf_file_t *fp, ctf_dtdef_t *dtd, unsigned char *t)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
{
|
|
|
|
ctf_dmdef_t *dmd = ctf_list_next (&dtd->dtd_u.dtu_members);
|
|
|
|
ctf_lmember_t ctlm;
|
|
|
|
|
|
|
|
for (; dmd != NULL; dmd = ctf_list_next (dmd))
|
|
|
|
{
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
ctf_lmember_t *copied;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
ctlm.ctlm_name = 0;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
ctlm.ctlm_type = (uint32_t) dmd->dmd_type;
|
|
|
|
ctlm.ctlm_offsethi = CTF_OFFSET_TO_LMEMHI (dmd->dmd_offset);
|
|
|
|
ctlm.ctlm_offsetlo = CTF_OFFSET_TO_LMEMLO (dmd->dmd_offset);
|
|
|
|
|
|
|
|
memcpy (t, &ctlm, sizeof (ctlm));
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
copied = (ctf_lmember_t *) t;
|
|
|
|
if (dmd->dmd_name)
|
|
|
|
ctf_str_add_ref (fp, dmd->dmd_name, &copied->ctlm_name);
|
|
|
|
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
t += sizeof (ctlm);
|
|
|
|
}
|
|
|
|
|
|
|
|
return t;
|
|
|
|
}
|
|
|
|
|
|
|
|
static unsigned char *
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
ctf_copy_emembers (ctf_file_t *fp, ctf_dtdef_t *dtd, unsigned char *t)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
{
|
|
|
|
ctf_dmdef_t *dmd = ctf_list_next (&dtd->dtd_u.dtu_members);
|
|
|
|
ctf_enum_t cte;
|
|
|
|
|
|
|
|
for (; dmd != NULL; dmd = ctf_list_next (dmd))
|
|
|
|
{
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
ctf_enum_t *copied;
|
|
|
|
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
cte.cte_value = dmd->dmd_value;
|
|
|
|
memcpy (t, &cte, sizeof (cte));
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
copied = (ctf_enum_t *) t;
|
|
|
|
ctf_str_add_ref (fp, dmd->dmd_name, &copied->cte_name);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
t += sizeof (cte);
|
|
|
|
}
|
|
|
|
|
|
|
|
return t;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Sort a newly-constructed static variable array. */
|
|
|
|
|
libctf: support getting strings from the ELF strtab
The CTF file format has always supported "external strtabs", which
internally are strtab offsets with their MSB on: such refs
get their strings from the strtab passed in at CTF file open time:
this is usually intended to be the ELF strtab, and that's what this
implementation is meant to support, though in theory the external
strtab could come from anywhere.
This commit adds support for these external strings in the ctf-string.c
strtab tracking layer. It's quite easy: we just add a field csa_offset
to the atoms table that tracks all strings: this field tracks the offset
of the string in the ELF strtab (with its MSB already on, courtesy of a
new macro CTF_SET_STID), and adds a new function that sets the
csa_offset to the specified offset (plus MSB). Then we just need to
avoid writing out strings to the internal strtab if they have csa_offset
set, and note that the internal strtab is shorter than it might
otherwise be.
(We could in theory save a little more time here by eschewing sorting
such strings, since we never actually write the strings out anywhere,
but that would mean storing them separately and it's just not worth the
complexity cost until profiling shows it's worth doing.)
We also have to go through a bit of extra effort at variable-sorting
time. This was previously using direct references to the internal
strtab: it couldn't use ctf_strptr or ctf_strraw because the new strtab
is not yet ready to put in its usual field (in a ctf_file_t that hasn't
even been allocated yet at this stage): but now we're using the external
strtab, this will no longer do because it'll be looking things up in the
wrong strtab, with disastrous results. Instead, pass the new internal
strtab in to a new ctf_strraw_explicit function which is just like
ctf_strraw except you can specify a ne winternal strtab to use.
But even now that it is using a new internal strtab, this is not quite
enough: it can't look up strings in the external strtab because ld
hasn't written it out yet, and when it does will write it straight to
disk. Instead, when we write the internal strtab, note all the offset
-> string mappings that we have noted belong in the *external* strtab to
a new "synthetic external strtab" dynhash, ctf_syn_ext_strtab, and look
in there at ctf_strraw time if it is set. This uses minimal extra
memory (because only strings in the external strtab that we actually use
are stored, and even those come straight out of the atoms table), but
let both variable sorting and name interning when ctf_bufopen is next
called work fine. (This also means that we don't need to filter out
spurious ECTF_STRTAB warnings from ctf_bufopen but can pass them back to
the caller, once we wrap ctf_bufopen so that we have a new internal
variant of ctf_bufopen etc that we can pass the synthetic external
strtab to. That error has been filtered out since the days of Solaris
libctf, which didn't try to handle the problem of getting external
strtabs right at construction time at all.)
v3: add the synthetic strtab and all associated machinery.
v5: fix tabdamage.
include/
* ctf.h (CTF_SET_STID): New.
libctf/
* ctf-impl.h (ctf_str_atom_t) <csa_offset>: New field.
(ctf_file_t) <ctf_syn_ext_strtab>: Likewise.
(ctf_str_add_ref): Name the last arg.
(ctf_str_add_external) New.
(ctf_str_add_strraw_explicit): Likewise.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
* ctf-string.c (ctf_strraw_explicit): Split from...
(ctf_strraw): ... here, with new support for ctf_syn_ext_strtab.
(ctf_str_add_ref_internal): Return the atom, not the
string.
(ctf_str_add): Adjust accordingly.
(ctf_str_add_ref): Likewise. Move up in the file.
(ctf_str_add_external): New: update the csa_offset.
(ctf_str_count_strtab): Only account for strings with no csa_offset
in the internal strtab length.
(ctf_str_write_strtab): If the csa_offset is set, update the
string's refs without writing the string out, and update the
ctf_syn_ext_strtab. Make OOM handling less ugly.
* ctf-create.c (struct ctf_sort_var_arg_cb): New.
(ctf_update): Handle failure to populate the strtab. Pass in the
new ctf_sort_var arg. Adjust for ctf_syn_ext_strtab addition.
Call ctf_simple_open_internal, not ctf_simple_open.
(ctf_sort_var): Call ctf_strraw_explicit rather than looking up
strings by hand.
* ctf-hash.c (ctf_hash_insert_type): Likewise (but using
ctf_strraw). Adjust to diagnose ECTF_STRTAB nonetheless.
* ctf-open.c (init_types): No longer filter out ECTF_STRTAB.
(ctf_file_close): Destroy the ctf_syn_ext_strtab.
(ctf_simple_open): Rename to, and reimplement as a wrapper around...
(ctf_simple_open_internal): ... this new function, which calls
ctf_bufopen_internal.
(ctf_bufopen): Rename to, and reimplement as a wrapper around...
(ctf_bufopen_internal): ... this new function, which sets
ctf_syn_ext_strtab.
2019-07-14 03:33:01 +08:00
|
|
|
typedef struct ctf_sort_var_arg_cb
|
|
|
|
{
|
|
|
|
ctf_file_t *fp;
|
|
|
|
ctf_strs_t *strtab;
|
|
|
|
} ctf_sort_var_arg_cb_t;
|
|
|
|
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
static int
|
libctf: support getting strings from the ELF strtab
The CTF file format has always supported "external strtabs", which
internally are strtab offsets with their MSB on: such refs
get their strings from the strtab passed in at CTF file open time:
this is usually intended to be the ELF strtab, and that's what this
implementation is meant to support, though in theory the external
strtab could come from anywhere.
This commit adds support for these external strings in the ctf-string.c
strtab tracking layer. It's quite easy: we just add a field csa_offset
to the atoms table that tracks all strings: this field tracks the offset
of the string in the ELF strtab (with its MSB already on, courtesy of a
new macro CTF_SET_STID), and adds a new function that sets the
csa_offset to the specified offset (plus MSB). Then we just need to
avoid writing out strings to the internal strtab if they have csa_offset
set, and note that the internal strtab is shorter than it might
otherwise be.
(We could in theory save a little more time here by eschewing sorting
such strings, since we never actually write the strings out anywhere,
but that would mean storing them separately and it's just not worth the
complexity cost until profiling shows it's worth doing.)
We also have to go through a bit of extra effort at variable-sorting
time. This was previously using direct references to the internal
strtab: it couldn't use ctf_strptr or ctf_strraw because the new strtab
is not yet ready to put in its usual field (in a ctf_file_t that hasn't
even been allocated yet at this stage): but now we're using the external
strtab, this will no longer do because it'll be looking things up in the
wrong strtab, with disastrous results. Instead, pass the new internal
strtab in to a new ctf_strraw_explicit function which is just like
ctf_strraw except you can specify a ne winternal strtab to use.
But even now that it is using a new internal strtab, this is not quite
enough: it can't look up strings in the external strtab because ld
hasn't written it out yet, and when it does will write it straight to
disk. Instead, when we write the internal strtab, note all the offset
-> string mappings that we have noted belong in the *external* strtab to
a new "synthetic external strtab" dynhash, ctf_syn_ext_strtab, and look
in there at ctf_strraw time if it is set. This uses minimal extra
memory (because only strings in the external strtab that we actually use
are stored, and even those come straight out of the atoms table), but
let both variable sorting and name interning when ctf_bufopen is next
called work fine. (This also means that we don't need to filter out
spurious ECTF_STRTAB warnings from ctf_bufopen but can pass them back to
the caller, once we wrap ctf_bufopen so that we have a new internal
variant of ctf_bufopen etc that we can pass the synthetic external
strtab to. That error has been filtered out since the days of Solaris
libctf, which didn't try to handle the problem of getting external
strtabs right at construction time at all.)
v3: add the synthetic strtab and all associated machinery.
v5: fix tabdamage.
include/
* ctf.h (CTF_SET_STID): New.
libctf/
* ctf-impl.h (ctf_str_atom_t) <csa_offset>: New field.
(ctf_file_t) <ctf_syn_ext_strtab>: Likewise.
(ctf_str_add_ref): Name the last arg.
(ctf_str_add_external) New.
(ctf_str_add_strraw_explicit): Likewise.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
* ctf-string.c (ctf_strraw_explicit): Split from...
(ctf_strraw): ... here, with new support for ctf_syn_ext_strtab.
(ctf_str_add_ref_internal): Return the atom, not the
string.
(ctf_str_add): Adjust accordingly.
(ctf_str_add_ref): Likewise. Move up in the file.
(ctf_str_add_external): New: update the csa_offset.
(ctf_str_count_strtab): Only account for strings with no csa_offset
in the internal strtab length.
(ctf_str_write_strtab): If the csa_offset is set, update the
string's refs without writing the string out, and update the
ctf_syn_ext_strtab. Make OOM handling less ugly.
* ctf-create.c (struct ctf_sort_var_arg_cb): New.
(ctf_update): Handle failure to populate the strtab. Pass in the
new ctf_sort_var arg. Adjust for ctf_syn_ext_strtab addition.
Call ctf_simple_open_internal, not ctf_simple_open.
(ctf_sort_var): Call ctf_strraw_explicit rather than looking up
strings by hand.
* ctf-hash.c (ctf_hash_insert_type): Likewise (but using
ctf_strraw). Adjust to diagnose ECTF_STRTAB nonetheless.
* ctf-open.c (init_types): No longer filter out ECTF_STRTAB.
(ctf_file_close): Destroy the ctf_syn_ext_strtab.
(ctf_simple_open): Rename to, and reimplement as a wrapper around...
(ctf_simple_open_internal): ... this new function, which calls
ctf_bufopen_internal.
(ctf_bufopen): Rename to, and reimplement as a wrapper around...
(ctf_bufopen_internal): ... this new function, which sets
ctf_syn_ext_strtab.
2019-07-14 03:33:01 +08:00
|
|
|
ctf_sort_var (const void *one_, const void *two_, void *arg_)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
{
|
|
|
|
const ctf_varent_t *one = one_;
|
|
|
|
const ctf_varent_t *two = two_;
|
libctf: support getting strings from the ELF strtab
The CTF file format has always supported "external strtabs", which
internally are strtab offsets with their MSB on: such refs
get their strings from the strtab passed in at CTF file open time:
this is usually intended to be the ELF strtab, and that's what this
implementation is meant to support, though in theory the external
strtab could come from anywhere.
This commit adds support for these external strings in the ctf-string.c
strtab tracking layer. It's quite easy: we just add a field csa_offset
to the atoms table that tracks all strings: this field tracks the offset
of the string in the ELF strtab (with its MSB already on, courtesy of a
new macro CTF_SET_STID), and adds a new function that sets the
csa_offset to the specified offset (plus MSB). Then we just need to
avoid writing out strings to the internal strtab if they have csa_offset
set, and note that the internal strtab is shorter than it might
otherwise be.
(We could in theory save a little more time here by eschewing sorting
such strings, since we never actually write the strings out anywhere,
but that would mean storing them separately and it's just not worth the
complexity cost until profiling shows it's worth doing.)
We also have to go through a bit of extra effort at variable-sorting
time. This was previously using direct references to the internal
strtab: it couldn't use ctf_strptr or ctf_strraw because the new strtab
is not yet ready to put in its usual field (in a ctf_file_t that hasn't
even been allocated yet at this stage): but now we're using the external
strtab, this will no longer do because it'll be looking things up in the
wrong strtab, with disastrous results. Instead, pass the new internal
strtab in to a new ctf_strraw_explicit function which is just like
ctf_strraw except you can specify a ne winternal strtab to use.
But even now that it is using a new internal strtab, this is not quite
enough: it can't look up strings in the external strtab because ld
hasn't written it out yet, and when it does will write it straight to
disk. Instead, when we write the internal strtab, note all the offset
-> string mappings that we have noted belong in the *external* strtab to
a new "synthetic external strtab" dynhash, ctf_syn_ext_strtab, and look
in there at ctf_strraw time if it is set. This uses minimal extra
memory (because only strings in the external strtab that we actually use
are stored, and even those come straight out of the atoms table), but
let both variable sorting and name interning when ctf_bufopen is next
called work fine. (This also means that we don't need to filter out
spurious ECTF_STRTAB warnings from ctf_bufopen but can pass them back to
the caller, once we wrap ctf_bufopen so that we have a new internal
variant of ctf_bufopen etc that we can pass the synthetic external
strtab to. That error has been filtered out since the days of Solaris
libctf, which didn't try to handle the problem of getting external
strtabs right at construction time at all.)
v3: add the synthetic strtab and all associated machinery.
v5: fix tabdamage.
include/
* ctf.h (CTF_SET_STID): New.
libctf/
* ctf-impl.h (ctf_str_atom_t) <csa_offset>: New field.
(ctf_file_t) <ctf_syn_ext_strtab>: Likewise.
(ctf_str_add_ref): Name the last arg.
(ctf_str_add_external) New.
(ctf_str_add_strraw_explicit): Likewise.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
* ctf-string.c (ctf_strraw_explicit): Split from...
(ctf_strraw): ... here, with new support for ctf_syn_ext_strtab.
(ctf_str_add_ref_internal): Return the atom, not the
string.
(ctf_str_add): Adjust accordingly.
(ctf_str_add_ref): Likewise. Move up in the file.
(ctf_str_add_external): New: update the csa_offset.
(ctf_str_count_strtab): Only account for strings with no csa_offset
in the internal strtab length.
(ctf_str_write_strtab): If the csa_offset is set, update the
string's refs without writing the string out, and update the
ctf_syn_ext_strtab. Make OOM handling less ugly.
* ctf-create.c (struct ctf_sort_var_arg_cb): New.
(ctf_update): Handle failure to populate the strtab. Pass in the
new ctf_sort_var arg. Adjust for ctf_syn_ext_strtab addition.
Call ctf_simple_open_internal, not ctf_simple_open.
(ctf_sort_var): Call ctf_strraw_explicit rather than looking up
strings by hand.
* ctf-hash.c (ctf_hash_insert_type): Likewise (but using
ctf_strraw). Adjust to diagnose ECTF_STRTAB nonetheless.
* ctf-open.c (init_types): No longer filter out ECTF_STRTAB.
(ctf_file_close): Destroy the ctf_syn_ext_strtab.
(ctf_simple_open): Rename to, and reimplement as a wrapper around...
(ctf_simple_open_internal): ... this new function, which calls
ctf_bufopen_internal.
(ctf_bufopen): Rename to, and reimplement as a wrapper around...
(ctf_bufopen_internal): ... this new function, which sets
ctf_syn_ext_strtab.
2019-07-14 03:33:01 +08:00
|
|
|
ctf_sort_var_arg_cb_t *arg = arg_;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
libctf: support getting strings from the ELF strtab
The CTF file format has always supported "external strtabs", which
internally are strtab offsets with their MSB on: such refs
get their strings from the strtab passed in at CTF file open time:
this is usually intended to be the ELF strtab, and that's what this
implementation is meant to support, though in theory the external
strtab could come from anywhere.
This commit adds support for these external strings in the ctf-string.c
strtab tracking layer. It's quite easy: we just add a field csa_offset
to the atoms table that tracks all strings: this field tracks the offset
of the string in the ELF strtab (with its MSB already on, courtesy of a
new macro CTF_SET_STID), and adds a new function that sets the
csa_offset to the specified offset (plus MSB). Then we just need to
avoid writing out strings to the internal strtab if they have csa_offset
set, and note that the internal strtab is shorter than it might
otherwise be.
(We could in theory save a little more time here by eschewing sorting
such strings, since we never actually write the strings out anywhere,
but that would mean storing them separately and it's just not worth the
complexity cost until profiling shows it's worth doing.)
We also have to go through a bit of extra effort at variable-sorting
time. This was previously using direct references to the internal
strtab: it couldn't use ctf_strptr or ctf_strraw because the new strtab
is not yet ready to put in its usual field (in a ctf_file_t that hasn't
even been allocated yet at this stage): but now we're using the external
strtab, this will no longer do because it'll be looking things up in the
wrong strtab, with disastrous results. Instead, pass the new internal
strtab in to a new ctf_strraw_explicit function which is just like
ctf_strraw except you can specify a ne winternal strtab to use.
But even now that it is using a new internal strtab, this is not quite
enough: it can't look up strings in the external strtab because ld
hasn't written it out yet, and when it does will write it straight to
disk. Instead, when we write the internal strtab, note all the offset
-> string mappings that we have noted belong in the *external* strtab to
a new "synthetic external strtab" dynhash, ctf_syn_ext_strtab, and look
in there at ctf_strraw time if it is set. This uses minimal extra
memory (because only strings in the external strtab that we actually use
are stored, and even those come straight out of the atoms table), but
let both variable sorting and name interning when ctf_bufopen is next
called work fine. (This also means that we don't need to filter out
spurious ECTF_STRTAB warnings from ctf_bufopen but can pass them back to
the caller, once we wrap ctf_bufopen so that we have a new internal
variant of ctf_bufopen etc that we can pass the synthetic external
strtab to. That error has been filtered out since the days of Solaris
libctf, which didn't try to handle the problem of getting external
strtabs right at construction time at all.)
v3: add the synthetic strtab and all associated machinery.
v5: fix tabdamage.
include/
* ctf.h (CTF_SET_STID): New.
libctf/
* ctf-impl.h (ctf_str_atom_t) <csa_offset>: New field.
(ctf_file_t) <ctf_syn_ext_strtab>: Likewise.
(ctf_str_add_ref): Name the last arg.
(ctf_str_add_external) New.
(ctf_str_add_strraw_explicit): Likewise.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
* ctf-string.c (ctf_strraw_explicit): Split from...
(ctf_strraw): ... here, with new support for ctf_syn_ext_strtab.
(ctf_str_add_ref_internal): Return the atom, not the
string.
(ctf_str_add): Adjust accordingly.
(ctf_str_add_ref): Likewise. Move up in the file.
(ctf_str_add_external): New: update the csa_offset.
(ctf_str_count_strtab): Only account for strings with no csa_offset
in the internal strtab length.
(ctf_str_write_strtab): If the csa_offset is set, update the
string's refs without writing the string out, and update the
ctf_syn_ext_strtab. Make OOM handling less ugly.
* ctf-create.c (struct ctf_sort_var_arg_cb): New.
(ctf_update): Handle failure to populate the strtab. Pass in the
new ctf_sort_var arg. Adjust for ctf_syn_ext_strtab addition.
Call ctf_simple_open_internal, not ctf_simple_open.
(ctf_sort_var): Call ctf_strraw_explicit rather than looking up
strings by hand.
* ctf-hash.c (ctf_hash_insert_type): Likewise (but using
ctf_strraw). Adjust to diagnose ECTF_STRTAB nonetheless.
* ctf-open.c (init_types): No longer filter out ECTF_STRTAB.
(ctf_file_close): Destroy the ctf_syn_ext_strtab.
(ctf_simple_open): Rename to, and reimplement as a wrapper around...
(ctf_simple_open_internal): ... this new function, which calls
ctf_bufopen_internal.
(ctf_bufopen): Rename to, and reimplement as a wrapper around...
(ctf_bufopen_internal): ... this new function, which sets
ctf_syn_ext_strtab.
2019-07-14 03:33:01 +08:00
|
|
|
return (strcmp (ctf_strraw_explicit (arg->fp, one->ctv_name, arg->strtab),
|
|
|
|
ctf_strraw_explicit (arg->fp, two->ctv_name, arg->strtab)));
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
}
|
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
/* Compatibility: just update the threshold for ctf_discard. */
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
int
|
|
|
|
ctf_update (ctf_file_t *fp)
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
{
|
|
|
|
if (!(fp->ctf_flags & LCTF_RDWR))
|
|
|
|
return (ctf_set_errno (fp, ECTF_RDONLY));
|
|
|
|
|
|
|
|
fp->ctf_dtoldid = fp->ctf_typemax;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* If the specified CTF container is writable and has been modified, reload this
|
|
|
|
container with the updated type definitions, ready for serialization. In
|
|
|
|
order to make this code and the rest of libctf as simple as possible, we
|
|
|
|
perform updates by taking the dynamic type definitions and creating an
|
|
|
|
in-memory CTF file containing the definitions, and then call
|
|
|
|
ctf_simple_open_internal() on it. We perform one extra trick here for the
|
|
|
|
benefit of callers and to keep our code simple: ctf_simple_open_internal()
|
|
|
|
will return a new ctf_file_t, but we want to keep the fp constant for the
|
|
|
|
caller, so after ctf_simple_open_internal() returns, we use memcpy to swap
|
|
|
|
the interior of the old and new ctf_file_t's, and then free the old. */
|
|
|
|
int
|
|
|
|
ctf_serialize (ctf_file_t *fp)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
{
|
|
|
|
ctf_file_t ofp, *nfp;
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
ctf_header_t hdr, *hdrp;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
ctf_dtdef_t *dtd;
|
|
|
|
ctf_dvdef_t *dvd;
|
|
|
|
ctf_varent_t *dvarents;
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
ctf_strs_writable_t strtab;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
unsigned char *t;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
unsigned long i;
|
|
|
|
size_t buf_size, type_size, nvars;
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
unsigned char *buf, *newbuf;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
int err;
|
|
|
|
|
|
|
|
if (!(fp->ctf_flags & LCTF_RDWR))
|
|
|
|
return (ctf_set_errno (fp, ECTF_RDONLY));
|
|
|
|
|
|
|
|
/* Update required? */
|
|
|
|
if (!(fp->ctf_flags & LCTF_DIRTY))
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
/* Fill in an initial CTF header. We will leave the label, object,
|
|
|
|
and function sections empty and only output a header, type section,
|
|
|
|
and string table. The type section begins at a 4-byte aligned
|
|
|
|
boundary past the CTF header itself (at relative offset zero). */
|
|
|
|
|
|
|
|
memset (&hdr, 0, sizeof (hdr));
|
|
|
|
hdr.cth_magic = CTF_MAGIC;
|
|
|
|
hdr.cth_version = CTF_VERSION;
|
|
|
|
|
|
|
|
/* Iterate through the dynamic type definition list and compute the
|
|
|
|
size of the CTF type section we will need to generate. */
|
|
|
|
|
|
|
|
for (type_size = 0, dtd = ctf_list_next (&fp->ctf_dtdefs);
|
|
|
|
dtd != NULL; dtd = ctf_list_next (dtd))
|
|
|
|
{
|
|
|
|
uint32_t kind = LCTF_INFO_KIND (fp, dtd->dtd_data.ctt_info);
|
|
|
|
uint32_t vlen = LCTF_INFO_VLEN (fp, dtd->dtd_data.ctt_info);
|
|
|
|
|
|
|
|
if (dtd->dtd_data.ctt_size != CTF_LSIZE_SENT)
|
|
|
|
type_size += sizeof (ctf_stype_t);
|
|
|
|
else
|
|
|
|
type_size += sizeof (ctf_type_t);
|
|
|
|
|
|
|
|
switch (kind)
|
|
|
|
{
|
|
|
|
case CTF_K_INTEGER:
|
|
|
|
case CTF_K_FLOAT:
|
|
|
|
type_size += sizeof (uint32_t);
|
|
|
|
break;
|
|
|
|
case CTF_K_ARRAY:
|
|
|
|
type_size += sizeof (ctf_array_t);
|
|
|
|
break;
|
|
|
|
case CTF_K_SLICE:
|
|
|
|
type_size += sizeof (ctf_slice_t);
|
|
|
|
break;
|
|
|
|
case CTF_K_FUNCTION:
|
|
|
|
type_size += sizeof (uint32_t) * (vlen + (vlen & 1));
|
|
|
|
break;
|
|
|
|
case CTF_K_STRUCT:
|
|
|
|
case CTF_K_UNION:
|
|
|
|
if (dtd->dtd_data.ctt_size < CTF_LSTRUCT_THRESH)
|
|
|
|
type_size += sizeof (ctf_member_t) * vlen;
|
|
|
|
else
|
|
|
|
type_size += sizeof (ctf_lmember_t) * vlen;
|
|
|
|
break;
|
|
|
|
case CTF_K_ENUM:
|
|
|
|
type_size += sizeof (ctf_enum_t) * vlen;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Computing the number of entries in the CTF variable section is much
|
|
|
|
simpler. */
|
|
|
|
|
|
|
|
for (nvars = 0, dvd = ctf_list_next (&fp->ctf_dvdefs);
|
|
|
|
dvd != NULL; dvd = ctf_list_next (dvd), nvars++);
|
|
|
|
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
/* Compute the size of the CTF buffer we need, sans only the string table,
|
|
|
|
then allocate a new buffer and memcpy the finished header to the start of
|
|
|
|
the buffer. (We will adjust this later with strtab length info.) */
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
|
|
|
hdr.cth_typeoff = hdr.cth_varoff + (nvars * sizeof (ctf_varent_t));
|
|
|
|
hdr.cth_stroff = hdr.cth_typeoff + type_size;
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
hdr.cth_strlen = 0;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
|
|
|
buf_size = sizeof (ctf_header_t) + hdr.cth_stroff + hdr.cth_strlen;
|
|
|
|
|
libctf: drop mmap()-based CTF data allocator
This allocator has the ostensible benefit that it lets us mprotect() the
memory used for CTF storage: but in exchange for this it adds
considerable complexity, since we have to track allocation sizes
ourselves for use at freeing time, note whether the data we are storing
was ctf_data_alloc()ed or not so we know if we can safely mprotect()
it... and while the mprotect()ing has found few bugs, it *has* been the
cause of more than one due to errors in all this tracking leading to us
mprotect()ing bits of the heap and stuff like that.
We are about to start composing CTF buffers from pieces so that we can
do usage-based optimizations on the strtab. This means we need
realloc(), which needs nonportable mremap() and *more* tracking of the
*original* allocation size, and the complexity and bureaucracy of all of
this is just too high for its negligible benefits.
Drop the whole thing and just use malloc() like everyone else. It knows
better than we do when it is safe to use mmap() under the covers,
anyway.
While we're at it, don't leak the entire buffer if ctf_compress_write()
fails to compress it.
libctf/
* ctf-subr.c (_PAGESIZE): Remove.
(ctf_data_alloc): Likewise.
(ctf_data_free): Likewise.
(ctf_data_protect): Likewise.
* ctf-impl.h: Remove declarations.
* ctf-create.c (ctf_update): No longer call ctf_data_protect: use
ctf_free, not ctf_data_free.
(ctf_compress_write): Use ctf_data_alloc, not ctf_alloc. Free
the buffer again on compression error.
* ctf-open.c (ctf_set_base): No longer track the size: call
ctf_free, not ctf_data_free.
(upgrade_types): Likewise. Call ctf_alloc, not ctf_data_alloc.
(ctf_bufopen): Likewise. No longer call ctf_data_protect.
2019-06-19 19:20:47 +08:00
|
|
|
if ((buf = malloc (buf_size)) == NULL)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return (ctf_set_errno (fp, EAGAIN));
|
|
|
|
|
|
|
|
memcpy (buf, &hdr, sizeof (ctf_header_t));
|
|
|
|
t = (unsigned char *) buf + sizeof (ctf_header_t) + hdr.cth_varoff;
|
|
|
|
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
hdrp = (ctf_header_t *) buf;
|
|
|
|
if ((fp->ctf_flags & LCTF_CHILD) && (fp->ctf_parname != NULL))
|
|
|
|
ctf_str_add_ref (fp, fp->ctf_parname, &hdrp->cth_parname);
|
libctf: allow the header to change between versions
libctf supports dynamic upgrading of the type table as file format
versions change, but before now has not supported changes to the CTF
header. Doing this is complicated by the baroque storage method used:
the CTF header is kept prepended to the rest of the CTF data, just as
when read from the file, and written out from there, and is
endian-flipped in place.
This makes accessing it needlessly hard and makes it almost impossible
to make the header larger if we add fields. The general storage
machinery around the malloced ctf pointer (the 'ctf_base') is also
overcomplicated: the pointer is sometimes malloced locally and sometimes
assigned from a parameter, so freeing it requires checking to see if
that parameter was used, needlessly coupling ctf_bufopen and
ctf_file_close together.
So split the header out into a new ctf_file_t.ctf_header, which is
written out explicitly: squeeze it out of the CTF buffer whenever we
reallocate it, and use ctf_file_t.ctf_buf to skip past the header when
we do not need to reallocate (when no upgrading or endian-flipping is
required). We now track whether the CTF base can be freed explicitly
via a new ctf_dynbase pointer which is non-NULL only when freeing is
possible.
With all this done, we can upgrade the header on the fly and add new
fields as desired, via a new upgrade_header function in ctf-open.
As with other forms of upgrading, libctf upgrades older headers
automatically to the latest supported version at open time.
For a first use of this field, we add a new string field cth_cuname, and
a corresponding setter/getter pair ctf_cuname_set and ctf_cuname: this
is used by debuggers to determine whether a CTF section's types relate
to a single compilation unit, or to all compilation units in the
program. (Types with ambiguous definitions in different CUs have only
one of these types placed in the top-level shared .ctf container: the
rest are placed in much smaller per-CU containers, which have the shared
container as their parent. Since CTF must be useful in the absence of
DWARF, we store the names of the relevant CUs ourselves, so the debugger
can look them up.)
v5: fix tabdamage.
include/
* ctf-api.h (ctf_cuname): New function.
(ctf_cuname_set): Likewise.
* ctf.h: Improve comment around upgrading, no longer
implying that v2 is the target of upgrades (it is v3 now).
(ctf_header_v2_t): New, old-format header for backward
compatibility.
(ctf_header_t): Add cth_cuname: this is the first of several
header changes in format v3.
libctf/
* ctf-impl.h (ctf_file_t): New fields ctf_header, ctf_dynbase,
ctf_cuname, ctf_dyncuname: ctf_base and ctf_buf are no longer const.
* ctf-open.c (ctf_set_base): Preserve the gap between ctf_buf and
ctf_base: do not assume that it is always sizeof (ctf_header_t).
Print out ctf_cuname: only print out ctf_parname if set.
(ctf_free_base): Removed, ctf_base is no longer freed: free
ctf_dynbase instead.
(ctf_set_version): Fix spacing.
(upgrade_header): New, in-place header upgrading.
(upgrade_types): Rename to...
(upgrade_types_v1): ... this. Free ctf_dynbase, not ctf_base. No
longer track old and new headers separately. No longer allow for
header sizes explicitly: squeeze the headers out on upgrade (they
are preserved in fp->ctf_header). Set ctf_dynbase, ctf_base and
ctf_buf explicitly. Use ctf_free, not ctf_free_base.
(upgrade_types): New, also handle ctf_parmax updating.
(flip_header): Flip ctf_cuname.
(flip_types): Flip BUF explicitly rather than deriving BUF from
BASE.
(ctf_bufopen): Store the header in fp->ctf_header. Correct minimum
required alignment of objtoff and funcoff. No longer store it in
the ctf_buf unless that buf is derived unmodified from the input.
Set ctf_dynbase where ctf_base is dynamically allocated. Drop locals
that duplicate fields in ctf_file: move allocation of ctf_file
further up instead. Call upgrade_header as needed. Move
version-specific ctf_parmax initialization into upgrade_types. More
concise error handling.
(ctf_file_close): No longer test for null pointers before freeing.
Free ctf_dyncuname, ctf_dynbase, and ctf_header. Do not call
ctf_free_base.
(ctf_cuname): New.
(ctf_cuname_set): New.
* ctf-create.c (ctf_update): Populate ctf_cuname.
(ctf_gzwrite): Write out the header explicitly. Remove obsolescent
comment.
(ctf_write): Likewise.
(ctf_compress_write): Get the header from ctf_header, not ctf_base.
Fix the compression length: fp->ctf_size never counted the CTF
header. Simplify the compress call accordingly.
2019-07-07 00:36:21 +08:00
|
|
|
if (fp->ctf_cuname != NULL)
|
|
|
|
ctf_str_add_ref (fp, fp->ctf_cuname, &hdrp->cth_cuname);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
/* Work over the variable list, translating everything into ctf_varent_t's and
|
|
|
|
prepping the string table. */
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
|
|
|
dvarents = (ctf_varent_t *) t;
|
|
|
|
for (i = 0, dvd = ctf_list_next (&fp->ctf_dvdefs); dvd != NULL;
|
|
|
|
dvd = ctf_list_next (dvd), i++)
|
|
|
|
{
|
|
|
|
ctf_varent_t *var = &dvarents[i];
|
|
|
|
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
ctf_str_add_ref (fp, dvd->dvd_name, &var->ctv_name);
|
2020-06-03 03:23:07 +08:00
|
|
|
var->ctv_type = (uint32_t) dvd->dvd_type;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
}
|
|
|
|
assert (i == nvars);
|
|
|
|
|
|
|
|
t += sizeof (ctf_varent_t) * nvars;
|
|
|
|
|
|
|
|
assert (t == (unsigned char *) buf + sizeof (ctf_header_t) + hdr.cth_typeoff);
|
|
|
|
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
/* We now take a final lap through the dynamic type definition list and copy
|
|
|
|
the appropriate type records to the output buffer, noting down the
|
|
|
|
strings as we go. */
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
|
|
|
for (dtd = ctf_list_next (&fp->ctf_dtdefs);
|
|
|
|
dtd != NULL; dtd = ctf_list_next (dtd))
|
|
|
|
{
|
|
|
|
uint32_t kind = LCTF_INFO_KIND (fp, dtd->dtd_data.ctt_info);
|
|
|
|
uint32_t vlen = LCTF_INFO_VLEN (fp, dtd->dtd_data.ctt_info);
|
|
|
|
|
|
|
|
ctf_array_t cta;
|
|
|
|
uint32_t encoding;
|
|
|
|
size_t len;
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
ctf_stype_t *copied;
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
const char *name;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
|
|
|
if (dtd->dtd_data.ctt_size != CTF_LSIZE_SENT)
|
|
|
|
len = sizeof (ctf_stype_t);
|
|
|
|
else
|
|
|
|
len = sizeof (ctf_type_t);
|
|
|
|
|
|
|
|
memcpy (t, &dtd->dtd_data, len);
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
copied = (ctf_stype_t *) t; /* name is at the start: constant offset. */
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
if (copied->ctt_name
|
|
|
|
&& (name = ctf_strraw (fp, copied->ctt_name)) != NULL)
|
|
|
|
ctf_str_add_ref (fp, name, &copied->ctt_name);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
t += len;
|
|
|
|
|
|
|
|
switch (kind)
|
|
|
|
{
|
|
|
|
case CTF_K_INTEGER:
|
|
|
|
case CTF_K_FLOAT:
|
|
|
|
if (kind == CTF_K_INTEGER)
|
|
|
|
{
|
|
|
|
encoding = CTF_INT_DATA (dtd->dtd_u.dtu_enc.cte_format,
|
|
|
|
dtd->dtd_u.dtu_enc.cte_offset,
|
|
|
|
dtd->dtd_u.dtu_enc.cte_bits);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
encoding = CTF_FP_DATA (dtd->dtd_u.dtu_enc.cte_format,
|
|
|
|
dtd->dtd_u.dtu_enc.cte_offset,
|
|
|
|
dtd->dtd_u.dtu_enc.cte_bits);
|
|
|
|
}
|
|
|
|
memcpy (t, &encoding, sizeof (encoding));
|
|
|
|
t += sizeof (encoding);
|
|
|
|
break;
|
|
|
|
|
|
|
|
case CTF_K_SLICE:
|
|
|
|
memcpy (t, &dtd->dtd_u.dtu_slice, sizeof (struct ctf_slice));
|
|
|
|
t += sizeof (struct ctf_slice);
|
|
|
|
break;
|
|
|
|
|
|
|
|
case CTF_K_ARRAY:
|
|
|
|
cta.cta_contents = (uint32_t) dtd->dtd_u.dtu_arr.ctr_contents;
|
|
|
|
cta.cta_index = (uint32_t) dtd->dtd_u.dtu_arr.ctr_index;
|
|
|
|
cta.cta_nelems = dtd->dtd_u.dtu_arr.ctr_nelems;
|
|
|
|
memcpy (t, &cta, sizeof (cta));
|
|
|
|
t += sizeof (cta);
|
|
|
|
break;
|
|
|
|
|
|
|
|
case CTF_K_FUNCTION:
|
|
|
|
{
|
|
|
|
uint32_t *argv = (uint32_t *) (uintptr_t) t;
|
|
|
|
uint32_t argc;
|
|
|
|
|
|
|
|
for (argc = 0; argc < vlen; argc++)
|
libctf, create: do not corrupt function types' arglists at insertion time
ctf_add_function assumes that function types' arglists are of type
ctf_id_t. Since they are CTF IDs, they are 32 bits wide, a uint32_t:
unfortunately ctf_id_t is a forward-compatible user-facing 64 bits wide,
and should never ever reach the CTF storage level.
All the CTF code other than ctf_add_function correctly assumes that
function arglists outside dynamic containers are 32 bits wide, so the
serialization machinery ends up cutting off half the arglist, corrupting
all args but the first (a good sign is a bunch of args of ID 0, the
unimplemented type, popping up).
Fix this by copying the arglist into place item by item, casting it
properly, at the same time as we validate the arg types. Fix the type
of the dtu_argv in the dynamic container and drop the now-unnecessary
cast in the serializer.
libctf/
* ctf-impl.h (ctf_dtdef_t) <dtu_argv>: Fix type.
* ctf-create.c (ctf_add_function): Check for unimplemented type
and populate at the same time. Populate one-by-one, not via
memcpy.
(ctf_serialize): Remove unnecessary cast.
* ctf-types.c (ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise. Fix comment typo.
2020-06-03 03:17:51 +08:00
|
|
|
*argv++ = dtd->dtd_u.dtu_argv[argc];
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
|
|
|
if (vlen & 1)
|
|
|
|
*argv++ = 0; /* Pad to 4-byte boundary. */
|
|
|
|
|
|
|
|
t = (unsigned char *) argv;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
case CTF_K_STRUCT:
|
|
|
|
case CTF_K_UNION:
|
|
|
|
if (dtd->dtd_data.ctt_size < CTF_LSTRUCT_THRESH)
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
t = ctf_copy_smembers (fp, dtd, t);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
else
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
t = ctf_copy_lmembers (fp, dtd, t);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
break;
|
|
|
|
|
|
|
|
case CTF_K_ENUM:
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
t = ctf_copy_emembers (fp, dtd, t);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
assert (t == (unsigned char *) buf + sizeof (ctf_header_t) + hdr.cth_stroff);
|
|
|
|
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
/* Construct the final string table and fill out all the string refs with the
|
|
|
|
final offsets. Then purge the refs list, because we're about to move this
|
|
|
|
strtab onto the end of the buf, invalidating all the offsets. */
|
|
|
|
strtab = ctf_str_write_strtab (fp);
|
|
|
|
ctf_str_purge_refs (fp);
|
|
|
|
|
libctf: support getting strings from the ELF strtab
The CTF file format has always supported "external strtabs", which
internally are strtab offsets with their MSB on: such refs
get their strings from the strtab passed in at CTF file open time:
this is usually intended to be the ELF strtab, and that's what this
implementation is meant to support, though in theory the external
strtab could come from anywhere.
This commit adds support for these external strings in the ctf-string.c
strtab tracking layer. It's quite easy: we just add a field csa_offset
to the atoms table that tracks all strings: this field tracks the offset
of the string in the ELF strtab (with its MSB already on, courtesy of a
new macro CTF_SET_STID), and adds a new function that sets the
csa_offset to the specified offset (plus MSB). Then we just need to
avoid writing out strings to the internal strtab if they have csa_offset
set, and note that the internal strtab is shorter than it might
otherwise be.
(We could in theory save a little more time here by eschewing sorting
such strings, since we never actually write the strings out anywhere,
but that would mean storing them separately and it's just not worth the
complexity cost until profiling shows it's worth doing.)
We also have to go through a bit of extra effort at variable-sorting
time. This was previously using direct references to the internal
strtab: it couldn't use ctf_strptr or ctf_strraw because the new strtab
is not yet ready to put in its usual field (in a ctf_file_t that hasn't
even been allocated yet at this stage): but now we're using the external
strtab, this will no longer do because it'll be looking things up in the
wrong strtab, with disastrous results. Instead, pass the new internal
strtab in to a new ctf_strraw_explicit function which is just like
ctf_strraw except you can specify a ne winternal strtab to use.
But even now that it is using a new internal strtab, this is not quite
enough: it can't look up strings in the external strtab because ld
hasn't written it out yet, and when it does will write it straight to
disk. Instead, when we write the internal strtab, note all the offset
-> string mappings that we have noted belong in the *external* strtab to
a new "synthetic external strtab" dynhash, ctf_syn_ext_strtab, and look
in there at ctf_strraw time if it is set. This uses minimal extra
memory (because only strings in the external strtab that we actually use
are stored, and even those come straight out of the atoms table), but
let both variable sorting and name interning when ctf_bufopen is next
called work fine. (This also means that we don't need to filter out
spurious ECTF_STRTAB warnings from ctf_bufopen but can pass them back to
the caller, once we wrap ctf_bufopen so that we have a new internal
variant of ctf_bufopen etc that we can pass the synthetic external
strtab to. That error has been filtered out since the days of Solaris
libctf, which didn't try to handle the problem of getting external
strtabs right at construction time at all.)
v3: add the synthetic strtab and all associated machinery.
v5: fix tabdamage.
include/
* ctf.h (CTF_SET_STID): New.
libctf/
* ctf-impl.h (ctf_str_atom_t) <csa_offset>: New field.
(ctf_file_t) <ctf_syn_ext_strtab>: Likewise.
(ctf_str_add_ref): Name the last arg.
(ctf_str_add_external) New.
(ctf_str_add_strraw_explicit): Likewise.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
* ctf-string.c (ctf_strraw_explicit): Split from...
(ctf_strraw): ... here, with new support for ctf_syn_ext_strtab.
(ctf_str_add_ref_internal): Return the atom, not the
string.
(ctf_str_add): Adjust accordingly.
(ctf_str_add_ref): Likewise. Move up in the file.
(ctf_str_add_external): New: update the csa_offset.
(ctf_str_count_strtab): Only account for strings with no csa_offset
in the internal strtab length.
(ctf_str_write_strtab): If the csa_offset is set, update the
string's refs without writing the string out, and update the
ctf_syn_ext_strtab. Make OOM handling less ugly.
* ctf-create.c (struct ctf_sort_var_arg_cb): New.
(ctf_update): Handle failure to populate the strtab. Pass in the
new ctf_sort_var arg. Adjust for ctf_syn_ext_strtab addition.
Call ctf_simple_open_internal, not ctf_simple_open.
(ctf_sort_var): Call ctf_strraw_explicit rather than looking up
strings by hand.
* ctf-hash.c (ctf_hash_insert_type): Likewise (but using
ctf_strraw). Adjust to diagnose ECTF_STRTAB nonetheless.
* ctf-open.c (init_types): No longer filter out ECTF_STRTAB.
(ctf_file_close): Destroy the ctf_syn_ext_strtab.
(ctf_simple_open): Rename to, and reimplement as a wrapper around...
(ctf_simple_open_internal): ... this new function, which calls
ctf_bufopen_internal.
(ctf_bufopen): Rename to, and reimplement as a wrapper around...
(ctf_bufopen_internal): ... this new function, which sets
ctf_syn_ext_strtab.
2019-07-14 03:33:01 +08:00
|
|
|
if (strtab.cts_strs == NULL)
|
|
|
|
{
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
free (buf);
|
libctf: support getting strings from the ELF strtab
The CTF file format has always supported "external strtabs", which
internally are strtab offsets with their MSB on: such refs
get their strings from the strtab passed in at CTF file open time:
this is usually intended to be the ELF strtab, and that's what this
implementation is meant to support, though in theory the external
strtab could come from anywhere.
This commit adds support for these external strings in the ctf-string.c
strtab tracking layer. It's quite easy: we just add a field csa_offset
to the atoms table that tracks all strings: this field tracks the offset
of the string in the ELF strtab (with its MSB already on, courtesy of a
new macro CTF_SET_STID), and adds a new function that sets the
csa_offset to the specified offset (plus MSB). Then we just need to
avoid writing out strings to the internal strtab if they have csa_offset
set, and note that the internal strtab is shorter than it might
otherwise be.
(We could in theory save a little more time here by eschewing sorting
such strings, since we never actually write the strings out anywhere,
but that would mean storing them separately and it's just not worth the
complexity cost until profiling shows it's worth doing.)
We also have to go through a bit of extra effort at variable-sorting
time. This was previously using direct references to the internal
strtab: it couldn't use ctf_strptr or ctf_strraw because the new strtab
is not yet ready to put in its usual field (in a ctf_file_t that hasn't
even been allocated yet at this stage): but now we're using the external
strtab, this will no longer do because it'll be looking things up in the
wrong strtab, with disastrous results. Instead, pass the new internal
strtab in to a new ctf_strraw_explicit function which is just like
ctf_strraw except you can specify a ne winternal strtab to use.
But even now that it is using a new internal strtab, this is not quite
enough: it can't look up strings in the external strtab because ld
hasn't written it out yet, and when it does will write it straight to
disk. Instead, when we write the internal strtab, note all the offset
-> string mappings that we have noted belong in the *external* strtab to
a new "synthetic external strtab" dynhash, ctf_syn_ext_strtab, and look
in there at ctf_strraw time if it is set. This uses minimal extra
memory (because only strings in the external strtab that we actually use
are stored, and even those come straight out of the atoms table), but
let both variable sorting and name interning when ctf_bufopen is next
called work fine. (This also means that we don't need to filter out
spurious ECTF_STRTAB warnings from ctf_bufopen but can pass them back to
the caller, once we wrap ctf_bufopen so that we have a new internal
variant of ctf_bufopen etc that we can pass the synthetic external
strtab to. That error has been filtered out since the days of Solaris
libctf, which didn't try to handle the problem of getting external
strtabs right at construction time at all.)
v3: add the synthetic strtab and all associated machinery.
v5: fix tabdamage.
include/
* ctf.h (CTF_SET_STID): New.
libctf/
* ctf-impl.h (ctf_str_atom_t) <csa_offset>: New field.
(ctf_file_t) <ctf_syn_ext_strtab>: Likewise.
(ctf_str_add_ref): Name the last arg.
(ctf_str_add_external) New.
(ctf_str_add_strraw_explicit): Likewise.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
* ctf-string.c (ctf_strraw_explicit): Split from...
(ctf_strraw): ... here, with new support for ctf_syn_ext_strtab.
(ctf_str_add_ref_internal): Return the atom, not the
string.
(ctf_str_add): Adjust accordingly.
(ctf_str_add_ref): Likewise. Move up in the file.
(ctf_str_add_external): New: update the csa_offset.
(ctf_str_count_strtab): Only account for strings with no csa_offset
in the internal strtab length.
(ctf_str_write_strtab): If the csa_offset is set, update the
string's refs without writing the string out, and update the
ctf_syn_ext_strtab. Make OOM handling less ugly.
* ctf-create.c (struct ctf_sort_var_arg_cb): New.
(ctf_update): Handle failure to populate the strtab. Pass in the
new ctf_sort_var arg. Adjust for ctf_syn_ext_strtab addition.
Call ctf_simple_open_internal, not ctf_simple_open.
(ctf_sort_var): Call ctf_strraw_explicit rather than looking up
strings by hand.
* ctf-hash.c (ctf_hash_insert_type): Likewise (but using
ctf_strraw). Adjust to diagnose ECTF_STRTAB nonetheless.
* ctf-open.c (init_types): No longer filter out ECTF_STRTAB.
(ctf_file_close): Destroy the ctf_syn_ext_strtab.
(ctf_simple_open): Rename to, and reimplement as a wrapper around...
(ctf_simple_open_internal): ... this new function, which calls
ctf_bufopen_internal.
(ctf_bufopen): Rename to, and reimplement as a wrapper around...
(ctf_bufopen_internal): ... this new function, which sets
ctf_syn_ext_strtab.
2019-07-14 03:33:01 +08:00
|
|
|
return (ctf_set_errno (fp, EAGAIN));
|
|
|
|
}
|
|
|
|
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
/* Now the string table is constructed, we can sort the buffer of
|
|
|
|
ctf_varent_t's. */
|
libctf: support getting strings from the ELF strtab
The CTF file format has always supported "external strtabs", which
internally are strtab offsets with their MSB on: such refs
get their strings from the strtab passed in at CTF file open time:
this is usually intended to be the ELF strtab, and that's what this
implementation is meant to support, though in theory the external
strtab could come from anywhere.
This commit adds support for these external strings in the ctf-string.c
strtab tracking layer. It's quite easy: we just add a field csa_offset
to the atoms table that tracks all strings: this field tracks the offset
of the string in the ELF strtab (with its MSB already on, courtesy of a
new macro CTF_SET_STID), and adds a new function that sets the
csa_offset to the specified offset (plus MSB). Then we just need to
avoid writing out strings to the internal strtab if they have csa_offset
set, and note that the internal strtab is shorter than it might
otherwise be.
(We could in theory save a little more time here by eschewing sorting
such strings, since we never actually write the strings out anywhere,
but that would mean storing them separately and it's just not worth the
complexity cost until profiling shows it's worth doing.)
We also have to go through a bit of extra effort at variable-sorting
time. This was previously using direct references to the internal
strtab: it couldn't use ctf_strptr or ctf_strraw because the new strtab
is not yet ready to put in its usual field (in a ctf_file_t that hasn't
even been allocated yet at this stage): but now we're using the external
strtab, this will no longer do because it'll be looking things up in the
wrong strtab, with disastrous results. Instead, pass the new internal
strtab in to a new ctf_strraw_explicit function which is just like
ctf_strraw except you can specify a ne winternal strtab to use.
But even now that it is using a new internal strtab, this is not quite
enough: it can't look up strings in the external strtab because ld
hasn't written it out yet, and when it does will write it straight to
disk. Instead, when we write the internal strtab, note all the offset
-> string mappings that we have noted belong in the *external* strtab to
a new "synthetic external strtab" dynhash, ctf_syn_ext_strtab, and look
in there at ctf_strraw time if it is set. This uses minimal extra
memory (because only strings in the external strtab that we actually use
are stored, and even those come straight out of the atoms table), but
let both variable sorting and name interning when ctf_bufopen is next
called work fine. (This also means that we don't need to filter out
spurious ECTF_STRTAB warnings from ctf_bufopen but can pass them back to
the caller, once we wrap ctf_bufopen so that we have a new internal
variant of ctf_bufopen etc that we can pass the synthetic external
strtab to. That error has been filtered out since the days of Solaris
libctf, which didn't try to handle the problem of getting external
strtabs right at construction time at all.)
v3: add the synthetic strtab and all associated machinery.
v5: fix tabdamage.
include/
* ctf.h (CTF_SET_STID): New.
libctf/
* ctf-impl.h (ctf_str_atom_t) <csa_offset>: New field.
(ctf_file_t) <ctf_syn_ext_strtab>: Likewise.
(ctf_str_add_ref): Name the last arg.
(ctf_str_add_external) New.
(ctf_str_add_strraw_explicit): Likewise.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
* ctf-string.c (ctf_strraw_explicit): Split from...
(ctf_strraw): ... here, with new support for ctf_syn_ext_strtab.
(ctf_str_add_ref_internal): Return the atom, not the
string.
(ctf_str_add): Adjust accordingly.
(ctf_str_add_ref): Likewise. Move up in the file.
(ctf_str_add_external): New: update the csa_offset.
(ctf_str_count_strtab): Only account for strings with no csa_offset
in the internal strtab length.
(ctf_str_write_strtab): If the csa_offset is set, update the
string's refs without writing the string out, and update the
ctf_syn_ext_strtab. Make OOM handling less ugly.
* ctf-create.c (struct ctf_sort_var_arg_cb): New.
(ctf_update): Handle failure to populate the strtab. Pass in the
new ctf_sort_var arg. Adjust for ctf_syn_ext_strtab addition.
Call ctf_simple_open_internal, not ctf_simple_open.
(ctf_sort_var): Call ctf_strraw_explicit rather than looking up
strings by hand.
* ctf-hash.c (ctf_hash_insert_type): Likewise (but using
ctf_strraw). Adjust to diagnose ECTF_STRTAB nonetheless.
* ctf-open.c (init_types): No longer filter out ECTF_STRTAB.
(ctf_file_close): Destroy the ctf_syn_ext_strtab.
(ctf_simple_open): Rename to, and reimplement as a wrapper around...
(ctf_simple_open_internal): ... this new function, which calls
ctf_bufopen_internal.
(ctf_bufopen): Rename to, and reimplement as a wrapper around...
(ctf_bufopen_internal): ... this new function, which sets
ctf_syn_ext_strtab.
2019-07-14 03:33:01 +08:00
|
|
|
ctf_sort_var_arg_cb_t sort_var_arg = { fp, (ctf_strs_t *) &strtab };
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
ctf_qsort_r (dvarents, nvars, sizeof (ctf_varent_t), ctf_sort_var,
|
libctf: support getting strings from the ELF strtab
The CTF file format has always supported "external strtabs", which
internally are strtab offsets with their MSB on: such refs
get their strings from the strtab passed in at CTF file open time:
this is usually intended to be the ELF strtab, and that's what this
implementation is meant to support, though in theory the external
strtab could come from anywhere.
This commit adds support for these external strings in the ctf-string.c
strtab tracking layer. It's quite easy: we just add a field csa_offset
to the atoms table that tracks all strings: this field tracks the offset
of the string in the ELF strtab (with its MSB already on, courtesy of a
new macro CTF_SET_STID), and adds a new function that sets the
csa_offset to the specified offset (plus MSB). Then we just need to
avoid writing out strings to the internal strtab if they have csa_offset
set, and note that the internal strtab is shorter than it might
otherwise be.
(We could in theory save a little more time here by eschewing sorting
such strings, since we never actually write the strings out anywhere,
but that would mean storing them separately and it's just not worth the
complexity cost until profiling shows it's worth doing.)
We also have to go through a bit of extra effort at variable-sorting
time. This was previously using direct references to the internal
strtab: it couldn't use ctf_strptr or ctf_strraw because the new strtab
is not yet ready to put in its usual field (in a ctf_file_t that hasn't
even been allocated yet at this stage): but now we're using the external
strtab, this will no longer do because it'll be looking things up in the
wrong strtab, with disastrous results. Instead, pass the new internal
strtab in to a new ctf_strraw_explicit function which is just like
ctf_strraw except you can specify a ne winternal strtab to use.
But even now that it is using a new internal strtab, this is not quite
enough: it can't look up strings in the external strtab because ld
hasn't written it out yet, and when it does will write it straight to
disk. Instead, when we write the internal strtab, note all the offset
-> string mappings that we have noted belong in the *external* strtab to
a new "synthetic external strtab" dynhash, ctf_syn_ext_strtab, and look
in there at ctf_strraw time if it is set. This uses minimal extra
memory (because only strings in the external strtab that we actually use
are stored, and even those come straight out of the atoms table), but
let both variable sorting and name interning when ctf_bufopen is next
called work fine. (This also means that we don't need to filter out
spurious ECTF_STRTAB warnings from ctf_bufopen but can pass them back to
the caller, once we wrap ctf_bufopen so that we have a new internal
variant of ctf_bufopen etc that we can pass the synthetic external
strtab to. That error has been filtered out since the days of Solaris
libctf, which didn't try to handle the problem of getting external
strtabs right at construction time at all.)
v3: add the synthetic strtab and all associated machinery.
v5: fix tabdamage.
include/
* ctf.h (CTF_SET_STID): New.
libctf/
* ctf-impl.h (ctf_str_atom_t) <csa_offset>: New field.
(ctf_file_t) <ctf_syn_ext_strtab>: Likewise.
(ctf_str_add_ref): Name the last arg.
(ctf_str_add_external) New.
(ctf_str_add_strraw_explicit): Likewise.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
* ctf-string.c (ctf_strraw_explicit): Split from...
(ctf_strraw): ... here, with new support for ctf_syn_ext_strtab.
(ctf_str_add_ref_internal): Return the atom, not the
string.
(ctf_str_add): Adjust accordingly.
(ctf_str_add_ref): Likewise. Move up in the file.
(ctf_str_add_external): New: update the csa_offset.
(ctf_str_count_strtab): Only account for strings with no csa_offset
in the internal strtab length.
(ctf_str_write_strtab): If the csa_offset is set, update the
string's refs without writing the string out, and update the
ctf_syn_ext_strtab. Make OOM handling less ugly.
* ctf-create.c (struct ctf_sort_var_arg_cb): New.
(ctf_update): Handle failure to populate the strtab. Pass in the
new ctf_sort_var arg. Adjust for ctf_syn_ext_strtab addition.
Call ctf_simple_open_internal, not ctf_simple_open.
(ctf_sort_var): Call ctf_strraw_explicit rather than looking up
strings by hand.
* ctf-hash.c (ctf_hash_insert_type): Likewise (but using
ctf_strraw). Adjust to diagnose ECTF_STRTAB nonetheless.
* ctf-open.c (init_types): No longer filter out ECTF_STRTAB.
(ctf_file_close): Destroy the ctf_syn_ext_strtab.
(ctf_simple_open): Rename to, and reimplement as a wrapper around...
(ctf_simple_open_internal): ... this new function, which calls
ctf_bufopen_internal.
(ctf_bufopen): Rename to, and reimplement as a wrapper around...
(ctf_bufopen_internal): ... this new function, which sets
ctf_syn_ext_strtab.
2019-07-14 03:33:01 +08:00
|
|
|
&sort_var_arg);
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
|
|
|
|
if ((newbuf = ctf_realloc (fp, buf, buf_size + strtab.cts_len)) == NULL)
|
|
|
|
{
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
free (buf);
|
|
|
|
free (strtab.cts_strs);
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
return (ctf_set_errno (fp, EAGAIN));
|
|
|
|
}
|
|
|
|
buf = newbuf;
|
|
|
|
memcpy (buf + buf_size, strtab.cts_strs, strtab.cts_len);
|
|
|
|
hdrp = (ctf_header_t *) buf;
|
|
|
|
hdrp->cth_strlen = strtab.cts_len;
|
|
|
|
buf_size += hdrp->cth_strlen;
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
free (strtab.cts_strs);
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
/* Finally, we are ready to ctf_simple_open() the new container. If this
|
|
|
|
is successful, we then switch nfp and fp and free the old container. */
|
|
|
|
|
libctf: support getting strings from the ELF strtab
The CTF file format has always supported "external strtabs", which
internally are strtab offsets with their MSB on: such refs
get their strings from the strtab passed in at CTF file open time:
this is usually intended to be the ELF strtab, and that's what this
implementation is meant to support, though in theory the external
strtab could come from anywhere.
This commit adds support for these external strings in the ctf-string.c
strtab tracking layer. It's quite easy: we just add a field csa_offset
to the atoms table that tracks all strings: this field tracks the offset
of the string in the ELF strtab (with its MSB already on, courtesy of a
new macro CTF_SET_STID), and adds a new function that sets the
csa_offset to the specified offset (plus MSB). Then we just need to
avoid writing out strings to the internal strtab if they have csa_offset
set, and note that the internal strtab is shorter than it might
otherwise be.
(We could in theory save a little more time here by eschewing sorting
such strings, since we never actually write the strings out anywhere,
but that would mean storing them separately and it's just not worth the
complexity cost until profiling shows it's worth doing.)
We also have to go through a bit of extra effort at variable-sorting
time. This was previously using direct references to the internal
strtab: it couldn't use ctf_strptr or ctf_strraw because the new strtab
is not yet ready to put in its usual field (in a ctf_file_t that hasn't
even been allocated yet at this stage): but now we're using the external
strtab, this will no longer do because it'll be looking things up in the
wrong strtab, with disastrous results. Instead, pass the new internal
strtab in to a new ctf_strraw_explicit function which is just like
ctf_strraw except you can specify a ne winternal strtab to use.
But even now that it is using a new internal strtab, this is not quite
enough: it can't look up strings in the external strtab because ld
hasn't written it out yet, and when it does will write it straight to
disk. Instead, when we write the internal strtab, note all the offset
-> string mappings that we have noted belong in the *external* strtab to
a new "synthetic external strtab" dynhash, ctf_syn_ext_strtab, and look
in there at ctf_strraw time if it is set. This uses minimal extra
memory (because only strings in the external strtab that we actually use
are stored, and even those come straight out of the atoms table), but
let both variable sorting and name interning when ctf_bufopen is next
called work fine. (This also means that we don't need to filter out
spurious ECTF_STRTAB warnings from ctf_bufopen but can pass them back to
the caller, once we wrap ctf_bufopen so that we have a new internal
variant of ctf_bufopen etc that we can pass the synthetic external
strtab to. That error has been filtered out since the days of Solaris
libctf, which didn't try to handle the problem of getting external
strtabs right at construction time at all.)
v3: add the synthetic strtab and all associated machinery.
v5: fix tabdamage.
include/
* ctf.h (CTF_SET_STID): New.
libctf/
* ctf-impl.h (ctf_str_atom_t) <csa_offset>: New field.
(ctf_file_t) <ctf_syn_ext_strtab>: Likewise.
(ctf_str_add_ref): Name the last arg.
(ctf_str_add_external) New.
(ctf_str_add_strraw_explicit): Likewise.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
* ctf-string.c (ctf_strraw_explicit): Split from...
(ctf_strraw): ... here, with new support for ctf_syn_ext_strtab.
(ctf_str_add_ref_internal): Return the atom, not the
string.
(ctf_str_add): Adjust accordingly.
(ctf_str_add_ref): Likewise. Move up in the file.
(ctf_str_add_external): New: update the csa_offset.
(ctf_str_count_strtab): Only account for strings with no csa_offset
in the internal strtab length.
(ctf_str_write_strtab): If the csa_offset is set, update the
string's refs without writing the string out, and update the
ctf_syn_ext_strtab. Make OOM handling less ugly.
* ctf-create.c (struct ctf_sort_var_arg_cb): New.
(ctf_update): Handle failure to populate the strtab. Pass in the
new ctf_sort_var arg. Adjust for ctf_syn_ext_strtab addition.
Call ctf_simple_open_internal, not ctf_simple_open.
(ctf_sort_var): Call ctf_strraw_explicit rather than looking up
strings by hand.
* ctf-hash.c (ctf_hash_insert_type): Likewise (but using
ctf_strraw). Adjust to diagnose ECTF_STRTAB nonetheless.
* ctf-open.c (init_types): No longer filter out ECTF_STRTAB.
(ctf_file_close): Destroy the ctf_syn_ext_strtab.
(ctf_simple_open): Rename to, and reimplement as a wrapper around...
(ctf_simple_open_internal): ... this new function, which calls
ctf_bufopen_internal.
(ctf_bufopen): Rename to, and reimplement as a wrapper around...
(ctf_bufopen_internal): ... this new function, which sets
ctf_syn_ext_strtab.
2019-07-14 03:33:01 +08:00
|
|
|
if ((nfp = ctf_simple_open_internal ((char *) buf, buf_size, NULL, 0,
|
|
|
|
0, NULL, 0, fp->ctf_syn_ext_strtab,
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
1, &err)) == NULL)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
{
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
free (buf);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return (ctf_set_errno (fp, err));
|
|
|
|
}
|
|
|
|
|
|
|
|
(void) ctf_setmodel (nfp, ctf_getmodel (fp));
|
|
|
|
|
libctf: sort out potential refcount loops
When you link TUs that contain conflicting types together, the resulting
CTF section is an archive containing many CTF dicts. These dicts appear
in ctf_link_outputs of the shared dict, with each ctf_import'ing that
shared dict. ctf_importing a dict bumps its refcount to stop it going
away while it's in use -- but if the shared dict (whose refcount is
bumped) has the child dict (doing the bumping) in its ctf_link_outputs,
we have a refcount loop, since the child dict only un-ctf_imports and
drops the parent's refcount when it is freed, but the child is only
freed when the parent's refcount falls to zero.
(In the future, this will be able to go wrong on the inputs too, when an
ld -r'ed deduplicated output with conflicts is relinked. Right now this
cannot happen because we don't ctf_import such dicts at all. This will
be fixed in a later commit in this series.)
Fix this by introducing an internal-use-only ctf_import_unref function
that imports a parent dict *witthout* bumping the parent's refcount, and
using it when we create per-CU outputs. This function is only safe to
use if you know the parent cannot go away while the child exists: but if
the parent *owns* the child, as here, this is necessarily true.
Record in the ctf_file_t whether a parent was imported via ctf_import or
ctf_import_unref, so that if you do another ctf_import later on (or a
ctf_import_unref) it can decide whether to drop the refcount of the
existing parent being replaced depending on which function you used to
import that one. Adjust ctf_serialize so that rather than doing a
ctf_import (which is wrong if the original import was
ctf_import_unref'fed), we just copy the parent field and refcount over
and forcibly flip the unref flag on on the old copy we are going to
discard.
ctf_file_close also needs a bit of tweaking to only close the parent if
it was not imported with ctf_import_unref: while we're at it, guard
against repeated closes with a refcount of zero and stop them causing
double-frees, even if destruction of things freed *inside*
ctf_file_close cause such recursion.
Verified no leaks or accesses to freed memory after all of this with
valgrind. (It was leak-happy before.)
libctf/
* ctf-impl.c (ctf_file_t) <ctf_parent_unreffed>: New.
(ctf_import_unref): New.
* ctf-open.c (ctf_file_close) Drop the refcount all the way to
zero. Don't recurse back in if the refcount is already zero.
(ctf_import): Check ctf_parent_unreffed before deciding whether
to close a pre-existing parent. Set it to zero.
(ctf_import_unreffed): New, as above, setting
ctf_parent_unreffed to 1.
* ctf-create.c (ctf_serialize): Do not ctf_import into the new
child: use direct assignment, and set unreffed on the new and
old children.
* ctf-link.c (ctf_create_per_cu): Import the parent using
ctf_import_unreffed.
2020-06-05 00:30:01 +08:00
|
|
|
nfp->ctf_parent = fp->ctf_parent;
|
|
|
|
nfp->ctf_parent_unreffed = fp->ctf_parent_unreffed;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
nfp->ctf_refcnt = fp->ctf_refcnt;
|
|
|
|
nfp->ctf_flags |= fp->ctf_flags & ~LCTF_DIRTY;
|
libctf: allow the header to change between versions
libctf supports dynamic upgrading of the type table as file format
versions change, but before now has not supported changes to the CTF
header. Doing this is complicated by the baroque storage method used:
the CTF header is kept prepended to the rest of the CTF data, just as
when read from the file, and written out from there, and is
endian-flipped in place.
This makes accessing it needlessly hard and makes it almost impossible
to make the header larger if we add fields. The general storage
machinery around the malloced ctf pointer (the 'ctf_base') is also
overcomplicated: the pointer is sometimes malloced locally and sometimes
assigned from a parameter, so freeing it requires checking to see if
that parameter was used, needlessly coupling ctf_bufopen and
ctf_file_close together.
So split the header out into a new ctf_file_t.ctf_header, which is
written out explicitly: squeeze it out of the CTF buffer whenever we
reallocate it, and use ctf_file_t.ctf_buf to skip past the header when
we do not need to reallocate (when no upgrading or endian-flipping is
required). We now track whether the CTF base can be freed explicitly
via a new ctf_dynbase pointer which is non-NULL only when freeing is
possible.
With all this done, we can upgrade the header on the fly and add new
fields as desired, via a new upgrade_header function in ctf-open.
As with other forms of upgrading, libctf upgrades older headers
automatically to the latest supported version at open time.
For a first use of this field, we add a new string field cth_cuname, and
a corresponding setter/getter pair ctf_cuname_set and ctf_cuname: this
is used by debuggers to determine whether a CTF section's types relate
to a single compilation unit, or to all compilation units in the
program. (Types with ambiguous definitions in different CUs have only
one of these types placed in the top-level shared .ctf container: the
rest are placed in much smaller per-CU containers, which have the shared
container as their parent. Since CTF must be useful in the absence of
DWARF, we store the names of the relevant CUs ourselves, so the debugger
can look them up.)
v5: fix tabdamage.
include/
* ctf-api.h (ctf_cuname): New function.
(ctf_cuname_set): Likewise.
* ctf.h: Improve comment around upgrading, no longer
implying that v2 is the target of upgrades (it is v3 now).
(ctf_header_v2_t): New, old-format header for backward
compatibility.
(ctf_header_t): Add cth_cuname: this is the first of several
header changes in format v3.
libctf/
* ctf-impl.h (ctf_file_t): New fields ctf_header, ctf_dynbase,
ctf_cuname, ctf_dyncuname: ctf_base and ctf_buf are no longer const.
* ctf-open.c (ctf_set_base): Preserve the gap between ctf_buf and
ctf_base: do not assume that it is always sizeof (ctf_header_t).
Print out ctf_cuname: only print out ctf_parname if set.
(ctf_free_base): Removed, ctf_base is no longer freed: free
ctf_dynbase instead.
(ctf_set_version): Fix spacing.
(upgrade_header): New, in-place header upgrading.
(upgrade_types): Rename to...
(upgrade_types_v1): ... this. Free ctf_dynbase, not ctf_base. No
longer track old and new headers separately. No longer allow for
header sizes explicitly: squeeze the headers out on upgrade (they
are preserved in fp->ctf_header). Set ctf_dynbase, ctf_base and
ctf_buf explicitly. Use ctf_free, not ctf_free_base.
(upgrade_types): New, also handle ctf_parmax updating.
(flip_header): Flip ctf_cuname.
(flip_types): Flip BUF explicitly rather than deriving BUF from
BASE.
(ctf_bufopen): Store the header in fp->ctf_header. Correct minimum
required alignment of objtoff and funcoff. No longer store it in
the ctf_buf unless that buf is derived unmodified from the input.
Set ctf_dynbase where ctf_base is dynamically allocated. Drop locals
that duplicate fields in ctf_file: move allocation of ctf_file
further up instead. Call upgrade_header as needed. Move
version-specific ctf_parmax initialization into upgrade_types. More
concise error handling.
(ctf_file_close): No longer test for null pointers before freeing.
Free ctf_dyncuname, ctf_dynbase, and ctf_header. Do not call
ctf_free_base.
(ctf_cuname): New.
(ctf_cuname_set): New.
* ctf-create.c (ctf_update): Populate ctf_cuname.
(ctf_gzwrite): Write out the header explicitly. Remove obsolescent
comment.
(ctf_write): Likewise.
(ctf_compress_write): Get the header from ctf_header, not ctf_base.
Fix the compression length: fp->ctf_size never counted the CTF
header. Simplify the compress call accordingly.
2019-07-07 00:36:21 +08:00
|
|
|
if (nfp->ctf_dynbase == NULL)
|
|
|
|
nfp->ctf_dynbase = buf; /* Make sure buf is freed on close. */
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
nfp->ctf_dthash = fp->ctf_dthash;
|
|
|
|
nfp->ctf_dtdefs = fp->ctf_dtdefs;
|
|
|
|
nfp->ctf_dvhash = fp->ctf_dvhash;
|
|
|
|
nfp->ctf_dvdefs = fp->ctf_dvdefs;
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
nfp->ctf_dtoldid = fp->ctf_dtoldid;
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
nfp->ctf_add_processing = fp->ctf_add_processing;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
nfp->ctf_snapshots = fp->ctf_snapshots + 1;
|
|
|
|
nfp->ctf_specific = fp->ctf_specific;
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
nfp->ctf_ptrtab = fp->ctf_ptrtab;
|
|
|
|
nfp->ctf_ptrtab_len = fp->ctf_ptrtab_len;
|
libctf: add the ctf_link machinery
This is the start of work on the core of the linking mechanism for CTF
sections. This commit handles the type and string sections.
The linker calls these functions in sequence:
ctf_link_add_ctf: to add each CTF section in the input in turn to a
newly-created ctf_file_t (which will appear in the output, and which
itself will become the shared parent that contains types that all
TUs have in common (in all link modes) and all types that do not
have conflicting definitions between types (by default). Input files
that are themselves products of ld -r are supported, though this is
not heavily tested yet.
ctf_link: called once all input files are added to merge the types in
all the input containers into the output container, eliminating
duplicates.
ctf_link_add_strtab: called once the ELF string table is finalized and
all its offsets are known, this calls a callback provided by the
linker which returns the string content and offset of every string in
the ELF strtab in turn: all these strings which appear in the input
CTF strtab are eliminated from it in favour of the ELF strtab:
equally, any strings that only appear in the input strtab will
reappear in the internal CTF strtab of the output.
ctf_link_shuffle_syms (not yet implemented): called once the ELF symtab
is finalized, this calls a callback provided by the linker which
returns information on every symbol in turn as a ctf_link_sym_t. This
is then used to shuffle the function info and data object sections in
the CTF section into symbol table order, eliminating the index
sections which map those sections to symbol names before that point.
Currently just returns ECTF_NOTYET.
ctf_link_write: Returns a buffer containing either a serialized
ctf_file_t (if there are no types with conflicting definitions in the
object files in the link) or a ctf_archive_t containing a large
ctf_file_t (the common types) and a bunch of small ones named after
individual CUs in which conflicting types are found (containing the
conflicting types, and all types that reference them). A threshold
size above which compression takes place is passed as one parameter.
(Currently, only gzip compression is supported, but I hope to add lzma
as well.)
Lifetime rules for this are simple: don't close the input CTF files
until you've called ctf_link for the last time. We do not assume
that symbols or strings passed in by the callback outlast the
call to ctf_link_add_strtab or ctf_link_shuffle_syms.
Right now, the duplicate elimination mechanism is the one already
present as part of the ctf_add_type function, and is not particularly
good: it misses numerous actual duplicates, and the conflicting-types
detection hardly ever reports that types conflict, even when they do
(one of them just tends to get silently dropped): it is also very slow.
This will all be fixed in the next few weeks, but the fix hardly touches
any of this code, and the linker does work without it, just not as
well as it otherwise might. (And when no CTF section is present,
there is no effect on performance, of course. So only people using
a trunk GCC with not-yet-committed patches will even notice. By the
time it gets upstream, things should be better.)
v3: Fix error handling.
v4: check for strdup failure.
v5: fix tabdamage.
include/
* ctf-api.h (struct ctf_link_sym): New, a symbol in flight to the
libctf linking machinery.
(CTF_LINK_SHARE_UNCONFLICTED): New.
(CTF_LINK_SHARE_DUPLICATED): New.
(ECTF_LINKADDEDLATE): New, replacing ECTF_UNUSED.
(ECTF_NOTYET): New, a 'not yet implemented' message.
(ctf_link_add_ctf): New, add an input file's CTF to the link.
(ctf_link): New, merge the type and string sections.
(ctf_link_strtab_string_f): New, callback for feeding strtab info.
(ctf_link_iter_symbol_f): New, callback for feeding symtab info.
(ctf_link_add_strtab): New, tell the CTF linker about the ELF
strtab's strings.
(ctf_link_shuffle_syms): New, ask the CTF linker to shuffle its
symbols into symtab order.
(ctf_link_write): New, ask the CTF linker to write the CTF out.
libctf/
* ctf-link.c: New file, linking of the string and type sections.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
* ctf-impl.h (ctf_file_t): New fields ctf_link_inputs,
ctf_link_outputs.
* ctf-create.c (ctf_update): Update accordingly.
* ctf-open.c (ctf_file_close): Likewise.
* ctf-error.c (_ctf_errlist): Updated with new errors.
2019-07-14 04:06:55 +08:00
|
|
|
nfp->ctf_link_inputs = fp->ctf_link_inputs;
|
|
|
|
nfp->ctf_link_outputs = fp->ctf_link_outputs;
|
libctf, ld, binutils: add textual error/warning reporting for libctf
This commit adds a long-missing piece of infrastructure to libctf: the
ability to report errors and warnings using all the power of printf,
rather than being restricted to one errno value. Internally, libctf
calls ctf_err_warn() to add errors and warnings to a list: a new
iterator ctf_errwarning_next() then consumes this list one by one and
hands it to the caller, which can free it. New errors and warnings are
added until the list is consumed by the caller or the ctf_file_t is
closed, so you can dump them at intervals. The caller can of course
choose to print only those warnings it wants. (I am not sure whether we
want objdump, readelf or ld to print warnings or not: right now I'm
printing them, but maybe we only want to print errors? This entirely
depends on whether warnings are voluminous things describing e.g. the
inability to emit single types because of name clashes or something.
There are no users of this infrastructure yet, so it's hard to say.)
There is no internationalization here yet, but this at least adds a
place where internationalization can be added, to one of
ctf_errwarning_next or ctf_err_warn.
We also provide a new ctf_assert() function which uses this
infrastructure to provide non-fatal assertion failures while emitting an
assert-like string to the caller: to save space and avoid needlessly
duplicating unchanging strings, the assertion test is inlined but the
print-things-out failure case is not. All assertions in libctf will be
converted to use this machinery in future commits and propagate
assertion-failure errors up, so that the linker in particular cannot be
killed by libctf assertion failures when it could perfectly well just
print warnings and drop the CTF section.
include/
* ctf-api.h (ECTF_INTERNAL): Adjust error text.
(ctf_errwarning_next): New.
libctf/
* ctf-impl.h (ctf_assert): New.
(ctf_err_warning_t): Likewise.
(ctf_file_t) <ctf_errs_warnings>: Likewise.
(ctf_err_warn): New prototype.
(ctf_assert_fail_internal): Likewise.
* ctf-inlines.h (ctf_assert_internal): Likewise.
* ctf-open.c (ctf_file_close): Free ctf_errs_warnings.
* ctf-create.c (ctf_serialize): Copy it on serialization.
* ctf-subr.c (ctf_err_warn): New, add an error/warning.
(ctf_errwarning_next): New iterator, free and pass back
errors/warnings in succession.
* libctf.ver (ctf_errwarning_next): Add.
ld/
* ldlang.c (lang_ctf_errs_warnings): New, print CTF errors
and warnings. Assert when libctf asserts.
(lang_merge_ctf): Call it.
(land_write_ctf): Likewise.
binutils/
* objdump.c (ctf_archive_member): Print CTF errors and warnings.
* readelf.c (dump_ctf_archive_member): Likewise.
2020-06-04 22:07:54 +08:00
|
|
|
nfp->ctf_errs_warnings = fp->ctf_errs_warnings;
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
nfp->ctf_str_prov_offset = fp->ctf_str_prov_offset;
|
libctf: support getting strings from the ELF strtab
The CTF file format has always supported "external strtabs", which
internally are strtab offsets with their MSB on: such refs
get their strings from the strtab passed in at CTF file open time:
this is usually intended to be the ELF strtab, and that's what this
implementation is meant to support, though in theory the external
strtab could come from anywhere.
This commit adds support for these external strings in the ctf-string.c
strtab tracking layer. It's quite easy: we just add a field csa_offset
to the atoms table that tracks all strings: this field tracks the offset
of the string in the ELF strtab (with its MSB already on, courtesy of a
new macro CTF_SET_STID), and adds a new function that sets the
csa_offset to the specified offset (plus MSB). Then we just need to
avoid writing out strings to the internal strtab if they have csa_offset
set, and note that the internal strtab is shorter than it might
otherwise be.
(We could in theory save a little more time here by eschewing sorting
such strings, since we never actually write the strings out anywhere,
but that would mean storing them separately and it's just not worth the
complexity cost until profiling shows it's worth doing.)
We also have to go through a bit of extra effort at variable-sorting
time. This was previously using direct references to the internal
strtab: it couldn't use ctf_strptr or ctf_strraw because the new strtab
is not yet ready to put in its usual field (in a ctf_file_t that hasn't
even been allocated yet at this stage): but now we're using the external
strtab, this will no longer do because it'll be looking things up in the
wrong strtab, with disastrous results. Instead, pass the new internal
strtab in to a new ctf_strraw_explicit function which is just like
ctf_strraw except you can specify a ne winternal strtab to use.
But even now that it is using a new internal strtab, this is not quite
enough: it can't look up strings in the external strtab because ld
hasn't written it out yet, and when it does will write it straight to
disk. Instead, when we write the internal strtab, note all the offset
-> string mappings that we have noted belong in the *external* strtab to
a new "synthetic external strtab" dynhash, ctf_syn_ext_strtab, and look
in there at ctf_strraw time if it is set. This uses minimal extra
memory (because only strings in the external strtab that we actually use
are stored, and even those come straight out of the atoms table), but
let both variable sorting and name interning when ctf_bufopen is next
called work fine. (This also means that we don't need to filter out
spurious ECTF_STRTAB warnings from ctf_bufopen but can pass them back to
the caller, once we wrap ctf_bufopen so that we have a new internal
variant of ctf_bufopen etc that we can pass the synthetic external
strtab to. That error has been filtered out since the days of Solaris
libctf, which didn't try to handle the problem of getting external
strtabs right at construction time at all.)
v3: add the synthetic strtab and all associated machinery.
v5: fix tabdamage.
include/
* ctf.h (CTF_SET_STID): New.
libctf/
* ctf-impl.h (ctf_str_atom_t) <csa_offset>: New field.
(ctf_file_t) <ctf_syn_ext_strtab>: Likewise.
(ctf_str_add_ref): Name the last arg.
(ctf_str_add_external) New.
(ctf_str_add_strraw_explicit): Likewise.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
* ctf-string.c (ctf_strraw_explicit): Split from...
(ctf_strraw): ... here, with new support for ctf_syn_ext_strtab.
(ctf_str_add_ref_internal): Return the atom, not the
string.
(ctf_str_add): Adjust accordingly.
(ctf_str_add_ref): Likewise. Move up in the file.
(ctf_str_add_external): New: update the csa_offset.
(ctf_str_count_strtab): Only account for strings with no csa_offset
in the internal strtab length.
(ctf_str_write_strtab): If the csa_offset is set, update the
string's refs without writing the string out, and update the
ctf_syn_ext_strtab. Make OOM handling less ugly.
* ctf-create.c (struct ctf_sort_var_arg_cb): New.
(ctf_update): Handle failure to populate the strtab. Pass in the
new ctf_sort_var arg. Adjust for ctf_syn_ext_strtab addition.
Call ctf_simple_open_internal, not ctf_simple_open.
(ctf_sort_var): Call ctf_strraw_explicit rather than looking up
strings by hand.
* ctf-hash.c (ctf_hash_insert_type): Likewise (but using
ctf_strraw). Adjust to diagnose ECTF_STRTAB nonetheless.
* ctf-open.c (init_types): No longer filter out ECTF_STRTAB.
(ctf_file_close): Destroy the ctf_syn_ext_strtab.
(ctf_simple_open): Rename to, and reimplement as a wrapper around...
(ctf_simple_open_internal): ... this new function, which calls
ctf_bufopen_internal.
(ctf_bufopen): Rename to, and reimplement as a wrapper around...
(ctf_bufopen_internal): ... this new function, which sets
ctf_syn_ext_strtab.
2019-07-14 03:33:01 +08:00
|
|
|
nfp->ctf_syn_ext_strtab = fp->ctf_syn_ext_strtab;
|
libctf, link: redo cu-mapping handling
Now a bunch of stuff that doesn't apply to ld or any normal use of
libctf, piled into one commit so that it's easier to ignore.
The cu-mapping machinery associates incoming compilation unit names with
outgoing names of CTF dictionaries that should correspond to them, for
non-gdb CTF consumers that would like to group multiple TUs into a
single child dict if conflicting types are found in it (the existing use
case is one kernel module, one child CTF dict, even if the kernel module
is composed of multiple CUs).
The upcoming deduplicator needs to track not only the mapping from
incoming CU name to outgoing dict name, but the inverse mapping from
outgoing dict name to incoming CU name, so it can work over every CTF
dict we might see in the output and link into it.
So rejig the ctf-link machinery to do that. Simultaneously (because
they are closely associated and were written at the same time), we add a
new CTF_LINK_EMPTY_CU_MAPPINGS flag to ctf_link, which tells the
ctf_link machinery to create empty child dicts for each outgoing CU
mapping even if no CUs that correspond to it exist in the link. This is
a bit (OK, quite a lot) of a waste of space, but some existing consumers
require it. (Nobody else should use it.)
Its value is not consecutive with existing CTF_LINK flag values because
we're about to add more flags that are conceptually closer to the
existing ones than this one is.
include/
* ctf-api.h (CTF_LINK_EMPTY_CU_MAPPINGS): New.
libctf/
* ctf-impl.h (ctf_file_t): Improve comments.
<ctf_link_cu_mapping>: Split into...
<ctf_link_in_cu_mapping>: ... this...
<ctf_link_out_cu_mapping>: ... and this.
* ctf-create.c (ctf_serialize): Adjust.
* ctf-open.c (ctf_file_close): Likewise.
* ctf-link.c (ctf_create_per_cu): Look things up in the
in_cu_mapping instead of the cu_mapping.
(ctf_link_add_cu_mapping): The deduplicating link will define
what happens if many FROMs share a TO.
(ctf_link_add_cu_mapping): Create in_cu_mapping and
out_cu_mapping. Do not create ctf_link_outputs here any more, or
create per-CU dicts here: they are already created when needed.
(ctf_link_one_variable): Log a debug message if we skip a
variable due to its type being concealed in a CU-mapped link.
(This is probably too common a case to make into a warning.)
(ctf_link): Create empty per-CU dicts if requested.
2020-06-06 00:36:16 +08:00
|
|
|
nfp->ctf_link_in_cu_mapping = fp->ctf_link_in_cu_mapping;
|
|
|
|
nfp->ctf_link_out_cu_mapping = fp->ctf_link_out_cu_mapping;
|
2019-07-14 04:31:26 +08:00
|
|
|
nfp->ctf_link_type_mapping = fp->ctf_link_type_mapping;
|
libctf: add CU-mapping machinery
Once the deduplicator is capable of actually detecting conflicting types
with the same name (i.e., not yet) we will place such conflicting types,
and types that depend on them, into CTF dictionaries that are the child
of the main dictionary we usually emit: currently, this will lead to the
.ctf section becoming a CTF archive rather than a single dictionary,
with the default-named archive member (_CTF_SECTION, or NULL) being the
main shared dictionary with most of the types in it.
By default, the sections are named after the compilation unit they come
from (complete path and all), with the cuname field in the CTF header
providing further evidence of the name without requiring the caller to
engage in tiresome parsing. But some callers may not wish the mapping
from input CU to output sub-dictionary to be purely CU-based.
The machinery here allows this to be freely changed, in two ways:
- callers can call ctf_link_add_cu_mapping to specify that a single
input compilation unit should have its types placed in some other CU
if they conflict: the CU will always be created, even if empty, so
the consuming program can depend on its existence. You can map
multiple input CUs to one output CU to force all their types to be
merged together: if some of *those* types conflict, the behaviour is
currently unspecified (the new deduplicator will specify it).
- callers can call ctf_link_set_memb_name_changer to provide a function
which is passed every CTF sub-dictionary name in turn (including
_CTF_SECTION) and can return a new name, or NULL if no change is
desired. The mapping from input to output names should not map two
input names to the same output name: if this happens, the two are not
merged but will result in an archive with two members with the same
name (technically valid, but it's hard to access the second
same-named member: you have to do an iteration over archive members).
This is used by the kernel's ctfarchive machinery (not yet upstream) to
encode CTF under member names like {module name}.ctf rather than
.ctf.CU, but it is anticipated that other large projects may wish to
have their own storage for CTF outside of .ctf sections and may wish to
have new naming schemes that suit their special-purpose consumers.
New in v3.
v4: check for strdup failure.
v5: fix tabdamage.
include/
* ctf-api.h (ctf_link_add_cu_mapping): New.
(ctf_link_memb_name_changer_f): New.
(ctf_link_set_memb_name_changer): New.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_link_cu_mappping>: New.
<ctf_link_memb_name_changer>: Likewise.
<ctf_link_memb_name_changer_arg>: Likewise.
* ctf-create.c (ctf_update): Update accordingly.
* ctf-open.c (ctf_file_close): Likewise.
* ctf-link.c (ctf_create_per_cu): Apply the cu mapping.
(ctf_link_add_cu_mapping): New.
(ctf_link_set_memb_name_changer): Likewise.
(ctf_change_parent_name): New.
(ctf_name_list_accum_cb_arg_t) <dynames>: New, storage for names
allocated by the caller's ctf_link_memb_name_changer.
<ndynames>: Likewise.
(ctf_accumulate_archive_names): Call the ctf_link_memb_name_changer.
(ctf_link_write): Likewise (for _CTF_SECTION only): also call
ctf_change_parent_name. Free any resulting names.
2019-07-20 21:44:44 +08:00
|
|
|
nfp->ctf_link_memb_name_changer = fp->ctf_link_memb_name_changer;
|
|
|
|
nfp->ctf_link_memb_name_changer_arg = fp->ctf_link_memb_name_changer_arg;
|
2020-06-06 01:15:26 +08:00
|
|
|
nfp->ctf_link_variable_filter = fp->ctf_link_variable_filter;
|
|
|
|
nfp->ctf_link_variable_filter_arg = fp->ctf_link_variable_filter_arg;
|
libctf, link: add lazy linking: clean up input members: err/warn cleanup
This rather large and intertwined pile of changes does three things:
First, it transitions from dprintf to ctf_err_warn for things the user might
care about: this one file is the major impetus for the ctf_err_warn
infrastructure, because things like file names are crucial in linker
error messages, and errno values are utterly incapable of
communicating them
Second, it stabilizes the ctf_link APIs: you can now call
ctf_link_add_ctf without a CTF argument (only a NAME), to lazily
ctf_open the file with the given NAME when needed, and close it as soon
as possible, to save memory. This is not an API change because a null
CTF argument was prohibited before now.
Since getting CTF directly from files uses ctf_open, passing in only a
NAME requires use of libctf, not libctf-nobfd. The linker's behaviour
is unchanged, as it still passes in a ctf_archive_t as before.
This also let us fix a leak: we were opening ctf_archives and their
containing ctf_files, then only closing the files and leaving the
archives open.
Third, this commit restructures the ctf_link_in_member argument used by
the CTF linking machinery and adjusts its users accordingly.
We drop two members:
- arcname, which is difficult to construct and then only used in error
messages (that were only dprintf()ed, so never seen!)
- share_mode, since we store the flags passed to ctf_link (including the
share mode) in a new ctf_file_t.ctf_link_flags to help dedup get hold
of it
We rename others whose existing names were fairly dreadful:
- done_main_member -> done_parent, using consistent terminology for .ctf
as the parent of all archive members
- main_input_fp -> in_fp_parent, likewise
- file_name -> in_file_name, likewise
We add one new member, cu_mapped.
Finally, we move the various frees of things like mapping table data to
the top-level ctf_link, since deduplicating links will want to do that
too.
include/
* ctf-api.h (ECTF_NEEDSBFD): New.
(ECTF_NERR): Adjust.
(ctf_link): Rename share_mode arg to flags.
libctf/
* Makefile.am: Set -DNOBFD=1 in libctf-nobfd, and =0 elsewhere.
* Makefile.in: Regenerated.
* ctf-impl.h (ctf_link_input_name): New.
(ctf_file_t) <ctf_link_flags>: New.
* ctf-create.c (ctf_serialize): Adjust accordingly.
* ctf-link.c: Define ctf_open as weak when PIC.
(ctf_arc_close_thunk): Remove unnecessary thunk.
(ctf_file_close_thunk): Likewise.
(ctf_link_input_name): New.
(ctf_link_input_t): New value of the ctf_file_t.ctf_link_input.
(ctf_link_input_close): Adjust accordingly.
(ctf_link_add_ctf_internal): New, split from...
(ctf_link_add_ctf): ... here. Return error if lazy loading of
CTF is not possible. Change to just call...
(ctf_link_add): ... this new function.
(ctf_link_add_cu_mapping): Transition to ctf_err_warn. Drop the
ctf_file_close_thunk.
(ctf_link_in_member_cb_arg_t) <file_name> Rename to...
<in_file_name>: ... this.
<arcname>: Drop.
<share_mode>: Likewise (migrated to ctf_link_flags).
<done_main_member>: Rename to...
<done_parent>: ... this.
<main_input_fp>: Rename to...
<in_fp_parent>: ... this.
<cu_mapped>: New.
(ctf_link_one_type): Adjuwt accordingly. Transition to
ctf_err_warn, removing a TODO.
(ctf_link_one_variable): Note a case too common to warn about.
Report in the debug stream if a cu-mapped link prevents addition
of a conflicting variable.
(ctf_link_one_input_archive_member): Adjust.
(ctf_link_lazy_open): New, open a CTF archive for linking when
needed.
(ctf_link_close_one_input_archive): New, close it again.
(ctf_link_one_input_archive): Adjust for lazy opening, member
renames, and ctf_err_warn transition. Move the
empty_link_type_mapping call to...
(ctf_link): ... here. Adjut for renamings and thunk removal.
Don't spuriously fail if some input contains no CTF data.
(ctf_link_write): ctf_err_warn transition.
* libctf.ver: Remove not-yet-stable comment.
2020-06-05 02:28:52 +08:00
|
|
|
nfp->ctf_link_flags = fp->ctf_link_flags;
|
2020-06-06 01:35:46 +08:00
|
|
|
nfp->ctf_dedup_atoms = fp->ctf_dedup_atoms;
|
|
|
|
nfp->ctf_dedup_atoms_alloc = fp->ctf_dedup_atoms_alloc;
|
|
|
|
memcpy (&nfp->ctf_dedup, &fp->ctf_dedup, sizeof (fp->ctf_dedup));
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
|
|
|
nfp->ctf_snapshot_lu = fp->ctf_snapshots;
|
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
memcpy (&nfp->ctf_lookups, fp->ctf_lookups, sizeof (fp->ctf_lookups));
|
|
|
|
nfp->ctf_structs = fp->ctf_structs;
|
|
|
|
nfp->ctf_unions = fp->ctf_unions;
|
|
|
|
nfp->ctf_enums = fp->ctf_enums;
|
|
|
|
nfp->ctf_names = fp->ctf_names;
|
|
|
|
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
fp->ctf_dthash = NULL;
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
ctf_str_free_atoms (nfp);
|
|
|
|
nfp->ctf_str_atoms = fp->ctf_str_atoms;
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
nfp->ctf_prov_strtab = fp->ctf_prov_strtab;
|
libctf: deduplicate and sort the string table
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
2019-06-27 20:51:10 +08:00
|
|
|
fp->ctf_str_atoms = NULL;
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
fp->ctf_prov_strtab = NULL;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
memset (&fp->ctf_dtdefs, 0, sizeof (ctf_list_t));
|
libctf, ld, binutils: add textual error/warning reporting for libctf
This commit adds a long-missing piece of infrastructure to libctf: the
ability to report errors and warnings using all the power of printf,
rather than being restricted to one errno value. Internally, libctf
calls ctf_err_warn() to add errors and warnings to a list: a new
iterator ctf_errwarning_next() then consumes this list one by one and
hands it to the caller, which can free it. New errors and warnings are
added until the list is consumed by the caller or the ctf_file_t is
closed, so you can dump them at intervals. The caller can of course
choose to print only those warnings it wants. (I am not sure whether we
want objdump, readelf or ld to print warnings or not: right now I'm
printing them, but maybe we only want to print errors? This entirely
depends on whether warnings are voluminous things describing e.g. the
inability to emit single types because of name clashes or something.
There are no users of this infrastructure yet, so it's hard to say.)
There is no internationalization here yet, but this at least adds a
place where internationalization can be added, to one of
ctf_errwarning_next or ctf_err_warn.
We also provide a new ctf_assert() function which uses this
infrastructure to provide non-fatal assertion failures while emitting an
assert-like string to the caller: to save space and avoid needlessly
duplicating unchanging strings, the assertion test is inlined but the
print-things-out failure case is not. All assertions in libctf will be
converted to use this machinery in future commits and propagate
assertion-failure errors up, so that the linker in particular cannot be
killed by libctf assertion failures when it could perfectly well just
print warnings and drop the CTF section.
include/
* ctf-api.h (ECTF_INTERNAL): Adjust error text.
(ctf_errwarning_next): New.
libctf/
* ctf-impl.h (ctf_assert): New.
(ctf_err_warning_t): Likewise.
(ctf_file_t) <ctf_errs_warnings>: Likewise.
(ctf_err_warn): New prototype.
(ctf_assert_fail_internal): Likewise.
* ctf-inlines.h (ctf_assert_internal): Likewise.
* ctf-open.c (ctf_file_close): Free ctf_errs_warnings.
* ctf-create.c (ctf_serialize): Copy it on serialization.
* ctf-subr.c (ctf_err_warn): New, add an error/warning.
(ctf_errwarning_next): New iterator, free and pass back
errors/warnings in succession.
* libctf.ver (ctf_errwarning_next): Add.
ld/
* ldlang.c (lang_ctf_errs_warnings): New, print CTF errors
and warnings. Assert when libctf asserts.
(lang_merge_ctf): Call it.
(land_write_ctf): Likewise.
binutils/
* objdump.c (ctf_archive_member): Print CTF errors and warnings.
* readelf.c (dump_ctf_archive_member): Likewise.
2020-06-04 22:07:54 +08:00
|
|
|
memset (&fp->ctf_errs_warnings, 0, sizeof (ctf_list_t));
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
fp->ctf_add_processing = NULL;
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
fp->ctf_ptrtab = NULL;
|
libctf: add the ctf_link machinery
This is the start of work on the core of the linking mechanism for CTF
sections. This commit handles the type and string sections.
The linker calls these functions in sequence:
ctf_link_add_ctf: to add each CTF section in the input in turn to a
newly-created ctf_file_t (which will appear in the output, and which
itself will become the shared parent that contains types that all
TUs have in common (in all link modes) and all types that do not
have conflicting definitions between types (by default). Input files
that are themselves products of ld -r are supported, though this is
not heavily tested yet.
ctf_link: called once all input files are added to merge the types in
all the input containers into the output container, eliminating
duplicates.
ctf_link_add_strtab: called once the ELF string table is finalized and
all its offsets are known, this calls a callback provided by the
linker which returns the string content and offset of every string in
the ELF strtab in turn: all these strings which appear in the input
CTF strtab are eliminated from it in favour of the ELF strtab:
equally, any strings that only appear in the input strtab will
reappear in the internal CTF strtab of the output.
ctf_link_shuffle_syms (not yet implemented): called once the ELF symtab
is finalized, this calls a callback provided by the linker which
returns information on every symbol in turn as a ctf_link_sym_t. This
is then used to shuffle the function info and data object sections in
the CTF section into symbol table order, eliminating the index
sections which map those sections to symbol names before that point.
Currently just returns ECTF_NOTYET.
ctf_link_write: Returns a buffer containing either a serialized
ctf_file_t (if there are no types with conflicting definitions in the
object files in the link) or a ctf_archive_t containing a large
ctf_file_t (the common types) and a bunch of small ones named after
individual CUs in which conflicting types are found (containing the
conflicting types, and all types that reference them). A threshold
size above which compression takes place is passed as one parameter.
(Currently, only gzip compression is supported, but I hope to add lzma
as well.)
Lifetime rules for this are simple: don't close the input CTF files
until you've called ctf_link for the last time. We do not assume
that symbols or strings passed in by the callback outlast the
call to ctf_link_add_strtab or ctf_link_shuffle_syms.
Right now, the duplicate elimination mechanism is the one already
present as part of the ctf_add_type function, and is not particularly
good: it misses numerous actual duplicates, and the conflicting-types
detection hardly ever reports that types conflict, even when they do
(one of them just tends to get silently dropped): it is also very slow.
This will all be fixed in the next few weeks, but the fix hardly touches
any of this code, and the linker does work without it, just not as
well as it otherwise might. (And when no CTF section is present,
there is no effect on performance, of course. So only people using
a trunk GCC with not-yet-committed patches will even notice. By the
time it gets upstream, things should be better.)
v3: Fix error handling.
v4: check for strdup failure.
v5: fix tabdamage.
include/
* ctf-api.h (struct ctf_link_sym): New, a symbol in flight to the
libctf linking machinery.
(CTF_LINK_SHARE_UNCONFLICTED): New.
(CTF_LINK_SHARE_DUPLICATED): New.
(ECTF_LINKADDEDLATE): New, replacing ECTF_UNUSED.
(ECTF_NOTYET): New, a 'not yet implemented' message.
(ctf_link_add_ctf): New, add an input file's CTF to the link.
(ctf_link): New, merge the type and string sections.
(ctf_link_strtab_string_f): New, callback for feeding strtab info.
(ctf_link_iter_symbol_f): New, callback for feeding symtab info.
(ctf_link_add_strtab): New, tell the CTF linker about the ELF
strtab's strings.
(ctf_link_shuffle_syms): New, ask the CTF linker to shuffle its
symbols into symtab order.
(ctf_link_write): New, ask the CTF linker to write the CTF out.
libctf/
* ctf-link.c: New file, linking of the string and type sections.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
* ctf-impl.h (ctf_file_t): New fields ctf_link_inputs,
ctf_link_outputs.
* ctf-create.c (ctf_update): Update accordingly.
* ctf-open.c (ctf_file_close): Likewise.
* ctf-error.c (_ctf_errlist): Updated with new errors.
2019-07-14 04:06:55 +08:00
|
|
|
fp->ctf_link_inputs = NULL;
|
|
|
|
fp->ctf_link_outputs = NULL;
|
libctf: support getting strings from the ELF strtab
The CTF file format has always supported "external strtabs", which
internally are strtab offsets with their MSB on: such refs
get their strings from the strtab passed in at CTF file open time:
this is usually intended to be the ELF strtab, and that's what this
implementation is meant to support, though in theory the external
strtab could come from anywhere.
This commit adds support for these external strings in the ctf-string.c
strtab tracking layer. It's quite easy: we just add a field csa_offset
to the atoms table that tracks all strings: this field tracks the offset
of the string in the ELF strtab (with its MSB already on, courtesy of a
new macro CTF_SET_STID), and adds a new function that sets the
csa_offset to the specified offset (plus MSB). Then we just need to
avoid writing out strings to the internal strtab if they have csa_offset
set, and note that the internal strtab is shorter than it might
otherwise be.
(We could in theory save a little more time here by eschewing sorting
such strings, since we never actually write the strings out anywhere,
but that would mean storing them separately and it's just not worth the
complexity cost until profiling shows it's worth doing.)
We also have to go through a bit of extra effort at variable-sorting
time. This was previously using direct references to the internal
strtab: it couldn't use ctf_strptr or ctf_strraw because the new strtab
is not yet ready to put in its usual field (in a ctf_file_t that hasn't
even been allocated yet at this stage): but now we're using the external
strtab, this will no longer do because it'll be looking things up in the
wrong strtab, with disastrous results. Instead, pass the new internal
strtab in to a new ctf_strraw_explicit function which is just like
ctf_strraw except you can specify a ne winternal strtab to use.
But even now that it is using a new internal strtab, this is not quite
enough: it can't look up strings in the external strtab because ld
hasn't written it out yet, and when it does will write it straight to
disk. Instead, when we write the internal strtab, note all the offset
-> string mappings that we have noted belong in the *external* strtab to
a new "synthetic external strtab" dynhash, ctf_syn_ext_strtab, and look
in there at ctf_strraw time if it is set. This uses minimal extra
memory (because only strings in the external strtab that we actually use
are stored, and even those come straight out of the atoms table), but
let both variable sorting and name interning when ctf_bufopen is next
called work fine. (This also means that we don't need to filter out
spurious ECTF_STRTAB warnings from ctf_bufopen but can pass them back to
the caller, once we wrap ctf_bufopen so that we have a new internal
variant of ctf_bufopen etc that we can pass the synthetic external
strtab to. That error has been filtered out since the days of Solaris
libctf, which didn't try to handle the problem of getting external
strtabs right at construction time at all.)
v3: add the synthetic strtab and all associated machinery.
v5: fix tabdamage.
include/
* ctf.h (CTF_SET_STID): New.
libctf/
* ctf-impl.h (ctf_str_atom_t) <csa_offset>: New field.
(ctf_file_t) <ctf_syn_ext_strtab>: Likewise.
(ctf_str_add_ref): Name the last arg.
(ctf_str_add_external) New.
(ctf_str_add_strraw_explicit): Likewise.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
* ctf-string.c (ctf_strraw_explicit): Split from...
(ctf_strraw): ... here, with new support for ctf_syn_ext_strtab.
(ctf_str_add_ref_internal): Return the atom, not the
string.
(ctf_str_add): Adjust accordingly.
(ctf_str_add_ref): Likewise. Move up in the file.
(ctf_str_add_external): New: update the csa_offset.
(ctf_str_count_strtab): Only account for strings with no csa_offset
in the internal strtab length.
(ctf_str_write_strtab): If the csa_offset is set, update the
string's refs without writing the string out, and update the
ctf_syn_ext_strtab. Make OOM handling less ugly.
* ctf-create.c (struct ctf_sort_var_arg_cb): New.
(ctf_update): Handle failure to populate the strtab. Pass in the
new ctf_sort_var arg. Adjust for ctf_syn_ext_strtab addition.
Call ctf_simple_open_internal, not ctf_simple_open.
(ctf_sort_var): Call ctf_strraw_explicit rather than looking up
strings by hand.
* ctf-hash.c (ctf_hash_insert_type): Likewise (but using
ctf_strraw). Adjust to diagnose ECTF_STRTAB nonetheless.
* ctf-open.c (init_types): No longer filter out ECTF_STRTAB.
(ctf_file_close): Destroy the ctf_syn_ext_strtab.
(ctf_simple_open): Rename to, and reimplement as a wrapper around...
(ctf_simple_open_internal): ... this new function, which calls
ctf_bufopen_internal.
(ctf_bufopen): Rename to, and reimplement as a wrapper around...
(ctf_bufopen_internal): ... this new function, which sets
ctf_syn_ext_strtab.
2019-07-14 03:33:01 +08:00
|
|
|
fp->ctf_syn_ext_strtab = NULL;
|
libctf, link: redo cu-mapping handling
Now a bunch of stuff that doesn't apply to ld or any normal use of
libctf, piled into one commit so that it's easier to ignore.
The cu-mapping machinery associates incoming compilation unit names with
outgoing names of CTF dictionaries that should correspond to them, for
non-gdb CTF consumers that would like to group multiple TUs into a
single child dict if conflicting types are found in it (the existing use
case is one kernel module, one child CTF dict, even if the kernel module
is composed of multiple CUs).
The upcoming deduplicator needs to track not only the mapping from
incoming CU name to outgoing dict name, but the inverse mapping from
outgoing dict name to incoming CU name, so it can work over every CTF
dict we might see in the output and link into it.
So rejig the ctf-link machinery to do that. Simultaneously (because
they are closely associated and were written at the same time), we add a
new CTF_LINK_EMPTY_CU_MAPPINGS flag to ctf_link, which tells the
ctf_link machinery to create empty child dicts for each outgoing CU
mapping even if no CUs that correspond to it exist in the link. This is
a bit (OK, quite a lot) of a waste of space, but some existing consumers
require it. (Nobody else should use it.)
Its value is not consecutive with existing CTF_LINK flag values because
we're about to add more flags that are conceptually closer to the
existing ones than this one is.
include/
* ctf-api.h (CTF_LINK_EMPTY_CU_MAPPINGS): New.
libctf/
* ctf-impl.h (ctf_file_t): Improve comments.
<ctf_link_cu_mapping>: Split into...
<ctf_link_in_cu_mapping>: ... this...
<ctf_link_out_cu_mapping>: ... and this.
* ctf-create.c (ctf_serialize): Adjust.
* ctf-open.c (ctf_file_close): Likewise.
* ctf-link.c (ctf_create_per_cu): Look things up in the
in_cu_mapping instead of the cu_mapping.
(ctf_link_add_cu_mapping): The deduplicating link will define
what happens if many FROMs share a TO.
(ctf_link_add_cu_mapping): Create in_cu_mapping and
out_cu_mapping. Do not create ctf_link_outputs here any more, or
create per-CU dicts here: they are already created when needed.
(ctf_link_one_variable): Log a debug message if we skip a
variable due to its type being concealed in a CU-mapped link.
(This is probably too common a case to make into a warning.)
(ctf_link): Create empty per-CU dicts if requested.
2020-06-06 00:36:16 +08:00
|
|
|
fp->ctf_link_in_cu_mapping = NULL;
|
|
|
|
fp->ctf_link_out_cu_mapping = NULL;
|
2019-07-14 04:31:26 +08:00
|
|
|
fp->ctf_link_type_mapping = NULL;
|
2020-06-06 01:35:46 +08:00
|
|
|
fp->ctf_dedup_atoms = NULL;
|
|
|
|
fp->ctf_dedup_atoms_alloc = NULL;
|
libctf: sort out potential refcount loops
When you link TUs that contain conflicting types together, the resulting
CTF section is an archive containing many CTF dicts. These dicts appear
in ctf_link_outputs of the shared dict, with each ctf_import'ing that
shared dict. ctf_importing a dict bumps its refcount to stop it going
away while it's in use -- but if the shared dict (whose refcount is
bumped) has the child dict (doing the bumping) in its ctf_link_outputs,
we have a refcount loop, since the child dict only un-ctf_imports and
drops the parent's refcount when it is freed, but the child is only
freed when the parent's refcount falls to zero.
(In the future, this will be able to go wrong on the inputs too, when an
ld -r'ed deduplicated output with conflicts is relinked. Right now this
cannot happen because we don't ctf_import such dicts at all. This will
be fixed in a later commit in this series.)
Fix this by introducing an internal-use-only ctf_import_unref function
that imports a parent dict *witthout* bumping the parent's refcount, and
using it when we create per-CU outputs. This function is only safe to
use if you know the parent cannot go away while the child exists: but if
the parent *owns* the child, as here, this is necessarily true.
Record in the ctf_file_t whether a parent was imported via ctf_import or
ctf_import_unref, so that if you do another ctf_import later on (or a
ctf_import_unref) it can decide whether to drop the refcount of the
existing parent being replaced depending on which function you used to
import that one. Adjust ctf_serialize so that rather than doing a
ctf_import (which is wrong if the original import was
ctf_import_unref'fed), we just copy the parent field and refcount over
and forcibly flip the unref flag on on the old copy we are going to
discard.
ctf_file_close also needs a bit of tweaking to only close the parent if
it was not imported with ctf_import_unref: while we're at it, guard
against repeated closes with a refcount of zero and stop them causing
double-frees, even if destruction of things freed *inside*
ctf_file_close cause such recursion.
Verified no leaks or accesses to freed memory after all of this with
valgrind. (It was leak-happy before.)
libctf/
* ctf-impl.c (ctf_file_t) <ctf_parent_unreffed>: New.
(ctf_import_unref): New.
* ctf-open.c (ctf_file_close) Drop the refcount all the way to
zero. Don't recurse back in if the refcount is already zero.
(ctf_import): Check ctf_parent_unreffed before deciding whether
to close a pre-existing parent. Set it to zero.
(ctf_import_unreffed): New, as above, setting
ctf_parent_unreffed to 1.
* ctf-create.c (ctf_serialize): Do not ctf_import into the new
child: use direct assignment, and set unreffed on the new and
old children.
* ctf-link.c (ctf_create_per_cu): Import the parent using
ctf_import_unreffed.
2020-06-05 00:30:01 +08:00
|
|
|
fp->ctf_parent_unreffed = 1;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
|
|
|
fp->ctf_dvhash = NULL;
|
|
|
|
memset (&fp->ctf_dvdefs, 0, sizeof (ctf_list_t));
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
memset (fp->ctf_lookups, 0, sizeof (fp->ctf_lookups));
|
2020-06-06 01:35:46 +08:00
|
|
|
memset (&fp->ctf_dedup, 0, sizeof (fp->ctf_dedup));
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
fp->ctf_structs.ctn_writable = NULL;
|
|
|
|
fp->ctf_unions.ctn_writable = NULL;
|
|
|
|
fp->ctf_enums.ctn_writable = NULL;
|
|
|
|
fp->ctf_names.ctn_writable = NULL;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
|
|
|
memcpy (&ofp, fp, sizeof (ctf_file_t));
|
|
|
|
memcpy (fp, nfp, sizeof (ctf_file_t));
|
|
|
|
memcpy (nfp, &ofp, sizeof (ctf_file_t));
|
|
|
|
|
|
|
|
nfp->ctf_refcnt = 1; /* Force nfp to be freed. */
|
|
|
|
ctf_file_close (nfp);
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
ctf_names_t *
|
|
|
|
ctf_name_table (ctf_file_t *fp, int kind)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
{
|
|
|
|
switch (kind)
|
|
|
|
{
|
|
|
|
case CTF_K_STRUCT:
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
return &fp->ctf_structs;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
case CTF_K_UNION:
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
return &fp->ctf_unions;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
case CTF_K_ENUM:
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
return &fp->ctf_enums;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
default:
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
return &fp->ctf_names;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2019-06-19 19:14:16 +08:00
|
|
|
int
|
2019-10-21 18:27:43 +08:00
|
|
|
ctf_dtd_insert (ctf_file_t *fp, ctf_dtdef_t *dtd, int flag, int kind)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
{
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
const char *name;
|
2019-06-19 19:14:16 +08:00
|
|
|
if (ctf_dynhash_insert (fp->ctf_dthash, (void *) dtd->dtd_type, dtd) < 0)
|
|
|
|
return -1;
|
|
|
|
|
2019-10-21 18:27:43 +08:00
|
|
|
if (flag == CTF_ADD_ROOT && dtd->dtd_data.ctt_name
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
&& (name = ctf_strraw (fp, dtd->dtd_data.ctt_name)) != NULL)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
{
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
if (ctf_dynhash_insert (ctf_name_table (fp, kind)->ctn_writable,
|
|
|
|
(char *) name, (void *) dtd->dtd_type) < 0)
|
|
|
|
{
|
|
|
|
ctf_dynhash_remove (fp->ctf_dthash, (void *) dtd->dtd_type);
|
|
|
|
return -1;
|
|
|
|
}
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
}
|
2019-06-19 19:14:16 +08:00
|
|
|
ctf_list_append (&fp->ctf_dtdefs, dtd);
|
|
|
|
return 0;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
ctf_dtd_delete (ctf_file_t *fp, ctf_dtdef_t *dtd)
|
|
|
|
{
|
|
|
|
ctf_dmdef_t *dmd, *nmd;
|
|
|
|
int kind = LCTF_INFO_KIND (fp, dtd->dtd_data.ctt_info);
|
libctf: create: forwards are always in the namespace of their referent
The C namespace a forward is located in is always the same as the
namespace of the corresponding complete type: 'struct foo' is in the
struct namespace and does not collide with, say, 'union foo'.
libctf allowed for this in many places, but inconsistently: in
particular, forward *addition* never allowed for this, and was interning
forwards in the default namespace, which is always wrong, since you can
only forward structs, unions and enums, all of which are in their own
namespaces in C.
Forward removal needs corresponding adjustment to remove the names form
the right namespace, as does ctf_rollback.
libctf/
* ctf-create.c (ctf_add_forward): Intern in the right namespace.
(ctf_dtd_delete): Remove correspondingly.
(ctf_rollback): Likewise.
2019-11-06 01:57:55 +08:00
|
|
|
int name_kind = kind;
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
const char *name;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
|
|
|
ctf_dynhash_remove (fp->ctf_dthash, (void *) dtd->dtd_type);
|
|
|
|
|
|
|
|
switch (kind)
|
|
|
|
{
|
|
|
|
case CTF_K_STRUCT:
|
|
|
|
case CTF_K_UNION:
|
|
|
|
case CTF_K_ENUM:
|
|
|
|
for (dmd = ctf_list_next (&dtd->dtd_u.dtu_members);
|
|
|
|
dmd != NULL; dmd = nmd)
|
|
|
|
{
|
|
|
|
if (dmd->dmd_name != NULL)
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
free (dmd->dmd_name);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
nmd = ctf_list_next (dmd);
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
free (dmd);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
}
|
|
|
|
break;
|
|
|
|
case CTF_K_FUNCTION:
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
free (dtd->dtd_u.dtu_argv);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
break;
|
libctf: create: forwards are always in the namespace of their referent
The C namespace a forward is located in is always the same as the
namespace of the corresponding complete type: 'struct foo' is in the
struct namespace and does not collide with, say, 'union foo'.
libctf allowed for this in many places, but inconsistently: in
particular, forward *addition* never allowed for this, and was interning
forwards in the default namespace, which is always wrong, since you can
only forward structs, unions and enums, all of which are in their own
namespaces in C.
Forward removal needs corresponding adjustment to remove the names form
the right namespace, as does ctf_rollback.
libctf/
* ctf-create.c (ctf_add_forward): Intern in the right namespace.
(ctf_dtd_delete): Remove correspondingly.
(ctf_rollback): Likewise.
2019-11-06 01:57:55 +08:00
|
|
|
case CTF_K_FORWARD:
|
|
|
|
name_kind = dtd->dtd_data.ctt_type;
|
|
|
|
break;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
}
|
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
if (dtd->dtd_data.ctt_name
|
2019-10-21 18:27:43 +08:00
|
|
|
&& (name = ctf_strraw (fp, dtd->dtd_data.ctt_name)) != NULL
|
|
|
|
&& LCTF_INFO_ISROOT (fp, dtd->dtd_data.ctt_info))
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
{
|
libctf: create: forwards are always in the namespace of their referent
The C namespace a forward is located in is always the same as the
namespace of the corresponding complete type: 'struct foo' is in the
struct namespace and does not collide with, say, 'union foo'.
libctf allowed for this in many places, but inconsistently: in
particular, forward *addition* never allowed for this, and was interning
forwards in the default namespace, which is always wrong, since you can
only forward structs, unions and enums, all of which are in their own
namespaces in C.
Forward removal needs corresponding adjustment to remove the names form
the right namespace, as does ctf_rollback.
libctf/
* ctf-create.c (ctf_add_forward): Intern in the right namespace.
(ctf_dtd_delete): Remove correspondingly.
(ctf_rollback): Likewise.
2019-11-06 01:57:55 +08:00
|
|
|
ctf_dynhash_remove (ctf_name_table (fp, name_kind)->ctn_writable,
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
name);
|
|
|
|
ctf_str_remove_ref (fp, name, &dtd->dtd_data.ctt_name);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
ctf_list_delete (&fp->ctf_dtdefs, dtd);
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
free (dtd);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
ctf_dtdef_t *
|
|
|
|
ctf_dtd_lookup (const ctf_file_t *fp, ctf_id_t type)
|
|
|
|
{
|
|
|
|
return (ctf_dtdef_t *) ctf_dynhash_lookup (fp->ctf_dthash, (void *) type);
|
|
|
|
}
|
|
|
|
|
|
|
|
ctf_dtdef_t *
|
|
|
|
ctf_dynamic_type (const ctf_file_t *fp, ctf_id_t id)
|
|
|
|
{
|
|
|
|
ctf_id_t idx;
|
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
if (!(fp->ctf_flags & LCTF_RDWR))
|
|
|
|
return NULL;
|
|
|
|
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
if ((fp->ctf_flags & LCTF_CHILD) && LCTF_TYPE_ISPARENT (fp, id))
|
|
|
|
fp = fp->ctf_parent;
|
|
|
|
|
|
|
|
idx = LCTF_TYPE_TO_INDEX(fp, id);
|
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
if ((unsigned long) idx <= fp->ctf_typemax)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return ctf_dtd_lookup (fp, id);
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
2019-06-19 19:14:16 +08:00
|
|
|
int
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
ctf_dvd_insert (ctf_file_t *fp, ctf_dvdef_t *dvd)
|
|
|
|
{
|
2019-06-19 19:14:16 +08:00
|
|
|
if (ctf_dynhash_insert (fp->ctf_dvhash, dvd->dvd_name, dvd) < 0)
|
|
|
|
return -1;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
ctf_list_append (&fp->ctf_dvdefs, dvd);
|
2019-06-19 19:14:16 +08:00
|
|
|
return 0;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
ctf_dvd_delete (ctf_file_t *fp, ctf_dvdef_t *dvd)
|
|
|
|
{
|
|
|
|
ctf_dynhash_remove (fp->ctf_dvhash, dvd->dvd_name);
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
free (dvd->dvd_name);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
|
|
|
ctf_list_delete (&fp->ctf_dvdefs, dvd);
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
free (dvd);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
ctf_dvdef_t *
|
|
|
|
ctf_dvd_lookup (const ctf_file_t *fp, const char *name)
|
|
|
|
{
|
|
|
|
return (ctf_dvdef_t *) ctf_dynhash_lookup (fp->ctf_dvhash, name);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Discard all of the dynamic type definitions and variable definitions that
|
|
|
|
have been added to the container since the last call to ctf_update(). We
|
|
|
|
locate such types by scanning the dtd list and deleting elements that have
|
|
|
|
type IDs greater than ctf_dtoldid, which is set by ctf_update(), above, and
|
|
|
|
by scanning the variable list and deleting elements that have update IDs
|
|
|
|
equal to the current value of the last-update snapshot count (indicating that
|
|
|
|
they were added after the most recent call to ctf_update()). */
|
|
|
|
int
|
|
|
|
ctf_discard (ctf_file_t *fp)
|
|
|
|
{
|
|
|
|
ctf_snapshot_id_t last_update =
|
|
|
|
{ fp->ctf_dtoldid,
|
|
|
|
fp->ctf_snapshot_lu + 1 };
|
|
|
|
|
|
|
|
/* Update required? */
|
|
|
|
if (!(fp->ctf_flags & LCTF_DIRTY))
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
return (ctf_rollback (fp, last_update));
|
|
|
|
}
|
|
|
|
|
|
|
|
ctf_snapshot_id_t
|
|
|
|
ctf_snapshot (ctf_file_t *fp)
|
|
|
|
{
|
|
|
|
ctf_snapshot_id_t snapid;
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
snapid.dtd_id = fp->ctf_typemax;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
snapid.snapshot_id = fp->ctf_snapshots++;
|
|
|
|
return snapid;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Like ctf_discard(), only discards everything after a particular ID. */
|
|
|
|
int
|
|
|
|
ctf_rollback (ctf_file_t *fp, ctf_snapshot_id_t id)
|
|
|
|
{
|
|
|
|
ctf_dtdef_t *dtd, *ntd;
|
|
|
|
ctf_dvdef_t *dvd, *nvd;
|
|
|
|
|
|
|
|
if (!(fp->ctf_flags & LCTF_RDWR))
|
|
|
|
return (ctf_set_errno (fp, ECTF_RDONLY));
|
|
|
|
|
|
|
|
if (fp->ctf_snapshot_lu >= id.snapshot_id)
|
|
|
|
return (ctf_set_errno (fp, ECTF_OVERROLLBACK));
|
|
|
|
|
|
|
|
for (dtd = ctf_list_next (&fp->ctf_dtdefs); dtd != NULL; dtd = ntd)
|
|
|
|
{
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
int kind;
|
|
|
|
const char *name;
|
|
|
|
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
ntd = ctf_list_next (dtd);
|
|
|
|
|
|
|
|
if (LCTF_TYPE_TO_INDEX (fp, dtd->dtd_type) <= id.dtd_id)
|
|
|
|
continue;
|
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
kind = LCTF_INFO_KIND (fp, dtd->dtd_data.ctt_info);
|
libctf: create: forwards are always in the namespace of their referent
The C namespace a forward is located in is always the same as the
namespace of the corresponding complete type: 'struct foo' is in the
struct namespace and does not collide with, say, 'union foo'.
libctf allowed for this in many places, but inconsistently: in
particular, forward *addition* never allowed for this, and was interning
forwards in the default namespace, which is always wrong, since you can
only forward structs, unions and enums, all of which are in their own
namespaces in C.
Forward removal needs corresponding adjustment to remove the names form
the right namespace, as does ctf_rollback.
libctf/
* ctf-create.c (ctf_add_forward): Intern in the right namespace.
(ctf_dtd_delete): Remove correspondingly.
(ctf_rollback): Likewise.
2019-11-06 01:57:55 +08:00
|
|
|
if (kind == CTF_K_FORWARD)
|
|
|
|
kind = dtd->dtd_data.ctt_type;
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
|
|
|
|
if (dtd->dtd_data.ctt_name
|
2019-10-21 18:27:43 +08:00
|
|
|
&& (name = ctf_strraw (fp, dtd->dtd_data.ctt_name)) != NULL
|
|
|
|
&& LCTF_INFO_ISROOT (fp, dtd->dtd_data.ctt_info))
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
{
|
|
|
|
ctf_dynhash_remove (ctf_name_table (fp, kind)->ctn_writable,
|
|
|
|
name);
|
|
|
|
ctf_str_remove_ref (fp, name, &dtd->dtd_data.ctt_name);
|
|
|
|
}
|
|
|
|
|
|
|
|
ctf_dynhash_remove (fp->ctf_dthash, (void *) dtd->dtd_type);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
ctf_dtd_delete (fp, dtd);
|
|
|
|
}
|
|
|
|
|
|
|
|
for (dvd = ctf_list_next (&fp->ctf_dvdefs); dvd != NULL; dvd = nvd)
|
|
|
|
{
|
|
|
|
nvd = ctf_list_next (dvd);
|
|
|
|
|
|
|
|
if (dvd->dvd_snapshots <= id.snapshot_id)
|
|
|
|
continue;
|
|
|
|
|
|
|
|
ctf_dvd_delete (fp, dvd);
|
|
|
|
}
|
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
fp->ctf_typemax = id.dtd_id;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
fp->ctf_snapshots = id.snapshot_id;
|
|
|
|
|
|
|
|
if (fp->ctf_snapshots == fp->ctf_snapshot_lu)
|
|
|
|
fp->ctf_flags &= ~LCTF_DIRTY;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static ctf_id_t
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
ctf_add_generic (ctf_file_t *fp, uint32_t flag, const char *name, int kind,
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
ctf_dtdef_t **rp)
|
|
|
|
{
|
|
|
|
ctf_dtdef_t *dtd;
|
|
|
|
ctf_id_t type;
|
|
|
|
|
|
|
|
if (flag != CTF_ADD_NONROOT && flag != CTF_ADD_ROOT)
|
|
|
|
return (ctf_set_errno (fp, EINVAL));
|
|
|
|
|
|
|
|
if (!(fp->ctf_flags & LCTF_RDWR))
|
|
|
|
return (ctf_set_errno (fp, ECTF_RDONLY));
|
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
if (LCTF_INDEX_TO_TYPE (fp, fp->ctf_typemax, 1) >= CTF_MAX_TYPE)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return (ctf_set_errno (fp, ECTF_FULL));
|
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
if (LCTF_INDEX_TO_TYPE (fp, fp->ctf_typemax, 1) == (CTF_MAX_PTYPE - 1))
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return (ctf_set_errno (fp, ECTF_FULL));
|
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
/* Make sure ptrtab always grows to be big enough for all types. */
|
|
|
|
if (ctf_grow_ptrtab (fp) < 0)
|
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
if ((dtd = malloc (sizeof (ctf_dtdef_t))) == NULL)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return (ctf_set_errno (fp, EAGAIN));
|
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
type = ++fp->ctf_typemax;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
type = LCTF_INDEX_TO_TYPE (fp, type, (fp->ctf_flags & LCTF_CHILD));
|
|
|
|
|
|
|
|
memset (dtd, 0, sizeof (ctf_dtdef_t));
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
dtd->dtd_data.ctt_name = ctf_str_add_ref (fp, name, &dtd->dtd_data.ctt_name);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
dtd->dtd_type = type;
|
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
if (dtd->dtd_data.ctt_name == 0 && name != NULL && name[0] != '\0')
|
|
|
|
{
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
free (dtd);
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
return (ctf_set_errno (fp, EAGAIN));
|
|
|
|
}
|
|
|
|
|
2019-10-21 18:27:43 +08:00
|
|
|
if (ctf_dtd_insert (fp, dtd, flag, kind) < 0)
|
2019-06-19 19:14:16 +08:00
|
|
|
{
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
free (dtd);
|
2019-06-19 19:14:16 +08:00
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
}
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
fp->ctf_flags |= LCTF_DIRTY;
|
|
|
|
|
|
|
|
*rp = dtd;
|
|
|
|
return type;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* When encoding integer sizes, we want to convert a byte count in the range
|
|
|
|
1-8 to the closest power of 2 (e.g. 3->4, 5->8, etc). The clp2() function
|
|
|
|
is a clever implementation from "Hacker's Delight" by Henry Warren, Jr. */
|
|
|
|
static size_t
|
|
|
|
clp2 (size_t x)
|
|
|
|
{
|
|
|
|
x--;
|
|
|
|
|
|
|
|
x |= (x >> 1);
|
|
|
|
x |= (x >> 2);
|
|
|
|
x |= (x >> 4);
|
|
|
|
x |= (x >> 8);
|
|
|
|
x |= (x >> 16);
|
|
|
|
|
|
|
|
return (x + 1);
|
|
|
|
}
|
|
|
|
|
2020-06-06 01:35:46 +08:00
|
|
|
ctf_id_t
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
ctf_add_encoded (ctf_file_t *fp, uint32_t flag,
|
|
|
|
const char *name, const ctf_encoding_t *ep, uint32_t kind)
|
|
|
|
{
|
|
|
|
ctf_dtdef_t *dtd;
|
|
|
|
ctf_id_t type;
|
|
|
|
|
|
|
|
if (ep == NULL)
|
|
|
|
return (ctf_set_errno (fp, EINVAL));
|
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
if ((type = ctf_add_generic (fp, flag, name, kind, &dtd)) == CTF_ERR)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
|
|
|
|
dtd->dtd_data.ctt_info = CTF_TYPE_INFO (kind, flag, 0);
|
Use CHAR_BIT instead of NBBY in libctf
On x86-64 Fedora 29, I tried to build a mingw-hosted gdb that targets
ppc-linux. You can do this with:
../binutils-gdb/configure --host=i686-w64-mingw32 --target=ppc-linux \
--disable-{binutils,gas,gold,gprof,ld}
The build failed with these errors in libctf:
In file included from ../../binutils-gdb/libctf/ctf-create.c:20:
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_encoded':
../../binutils-gdb/libctf/ctf-create.c:803:59: error: 'NBBY' undeclared (first use in this function)
dtd->dtd_data.ctt_size = clp2 (P2ROUNDUP (ep->cte_bits, NBBY) / NBBY);
^~~~
../../binutils-gdb/libctf/ctf-impl.h:254:42: note: in definition of macro 'P2ROUNDUP'
#define P2ROUNDUP(x, align) (-(-(x) & -(align)))
^~~~~
../../binutils-gdb/libctf/ctf-create.c:803:59: note: each undeclared identifier is reported only once for each function it appears in
dtd->dtd_data.ctt_size = clp2 (P2ROUNDUP (ep->cte_bits, NBBY) / NBBY);
^~~~
../../binutils-gdb/libctf/ctf-impl.h:254:42: note: in definition of macro 'P2ROUNDUP'
#define P2ROUNDUP(x, align) (-(-(x) & -(align)))
^~~~~
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_slice':
../../binutils-gdb/libctf/ctf-create.c:862:59: error: 'NBBY' undeclared (first use in this function)
dtd->dtd_data.ctt_size = clp2 (P2ROUNDUP (ep->cte_bits, NBBY) / NBBY);
^~~~
../../binutils-gdb/libctf/ctf-impl.h:254:42: note: in definition of macro 'P2ROUNDUP'
#define P2ROUNDUP(x, align) (-(-(x) & -(align)))
^~~~~
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_member_offset':
../../binutils-gdb/libctf/ctf-create.c:1341:21: error: 'NBBY' undeclared (first use in this function)
off += lsize * NBBY;
^~~~
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_type':
../../binutils-gdb/libctf/ctf-create.c:1822:16: warning: unknown conversion type character 'z' in format [-Wformat=]
ctf_dprintf ("Conflict for type %s against ID %lx: "
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../binutils-gdb/libctf/ctf-create.c:1823:35: note: format string is defined here
"union size differs, old %zi, new %zi\n",
^
../../binutils-gdb/libctf/ctf-create.c:1822:16: warning: unknown conversion type character 'z' in format [-Wformat=]
ctf_dprintf ("Conflict for type %s against ID %lx: "
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../binutils-gdb/libctf/ctf-create.c:1823:44: note: format string is defined here
"union size differs, old %zi, new %zi\n",
^
../../binutils-gdb/libctf/ctf-create.c:1822:16: warning: too many arguments for format [-Wformat-extra-args]
ctf_dprintf ("Conflict for type %s against ID %lx: "
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This patch fixes the actual errors in here. I did not try to fix the
printf warnings, though I think someone ought to.
Ok?
libctf/ChangeLog
2019-06-04 Tom Tromey <tromey@adacore.com>
* ctf-create.c (ctf_add_encoded, ctf_add_slice)
(ctf_add_member_offset): Use CHAR_BIT, not NBBY.
2019-06-05 02:16:57 +08:00
|
|
|
dtd->dtd_data.ctt_size = clp2 (P2ROUNDUP (ep->cte_bits, CHAR_BIT)
|
|
|
|
/ CHAR_BIT);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
dtd->dtd_u.dtu_enc = *ep;
|
|
|
|
|
|
|
|
return type;
|
|
|
|
}
|
|
|
|
|
2020-06-06 01:35:46 +08:00
|
|
|
ctf_id_t
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
ctf_add_reftype (ctf_file_t *fp, uint32_t flag, ctf_id_t ref, uint32_t kind)
|
|
|
|
{
|
|
|
|
ctf_dtdef_t *dtd;
|
|
|
|
ctf_id_t type;
|
|
|
|
ctf_file_t *tmp = fp;
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
int child = fp->ctf_flags & LCTF_CHILD;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
libctf: fix a number of build problems found on Solaris and NetBSD
- Use of nonportable <endian.h>
- Use of qsort_r
- Use of zlib without appropriate magic to pull in the binutils zlib
- Use of off64_t without checking (fixed by dropping the unused fields
that need off64_t entirely)
- signedness problems due to long being too short a type on 32-bit
platforms: ctf_id_t is now 'unsigned long', and CTF_ERR must be
used only for functions that return ctf_id_t
- One lingering use of bzero() and of <sys/errno.h>
All fixed, using code from gnulib where possible.
Relatedly, set cts_size in a couple of places it was missed
(string table and symbol table loading upon ctf_bfdopen()).
binutils/
* objdump.c (make_ctfsect): Drop cts_type, cts_flags, and
cts_offset.
* readelf.c (shdr_to_ctf_sect): Likewise.
include/
* ctf-api.h (ctf_sect_t): Drop cts_type, cts_flags, and cts_offset.
(ctf_id_t): This is now an unsigned type.
(CTF_ERR): Cast it to ctf_id_t. Note that it should only be used
for ctf_id_t-returning functions.
libctf/
* Makefile.am (ZLIB): New.
(ZLIBINC): Likewise.
(AM_CFLAGS): Use them.
(libctf_a_LIBADD): New, for LIBOBJS.
* configure.ac: Check for zlib, endian.h, and qsort_r.
* ctf-endian.h: New, providing htole64 and le64toh.
* swap.h: Code style fixes.
(bswap_identity_64): New.
* qsort_r.c: New, from gnulib (with one added #include).
* ctf-decls.h: New, providing a conditional qsort_r declaration,
and unconditional definitions of MIN and MAX.
* ctf-impl.h: Use it. Do not use <sys/errno.h>.
(ctf_set_errno): Now returns unsigned long.
* ctf-util.c (ctf_set_errno): Adjust here too.
* ctf-archive.c: Use ctf-endian.h.
(ctf_arc_open_by_offset): Use memset, not bzero. Drop cts_type,
cts_flags and cts_offset.
(ctf_arc_write): Drop debugging dependent on the size of off_t.
* ctf-create.c: Provide a definition of roundup if not defined.
(ctf_create): Drop cts_type, cts_flags and cts_offset.
(ctf_add_reftype): Do not check if type IDs are below zero.
(ctf_add_slice): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_member_offset): Cast error-returning ssize_t's to size_t
when known error-free. Drop CTF_ERR usage for functions returning
int.
(ctf_add_member_encoded): Drop CTF_ERR usage for functions returning
int.
(ctf_add_variable): Likewise.
(enumcmp): Likewise.
(enumadd): Likewise.
(membcmp): Likewise.
(ctf_add_type): Likewise. Cast error-returning ssize_t's to size_t
when known error-free.
* ctf-dump.c (ctf_is_slice): Drop CTF_ERR usage for functions
returning int: use CTF_ERR for functions returning ctf_type_id.
(ctf_dump_label): Likewise.
(ctf_dump_objts): Likewise.
* ctf-labels.c (ctf_label_topmost): Likewise.
(ctf_label_iter): Likewise.
(ctf_label_info): Likewise.
* ctf-lookup.c (ctf_func_args): Likewise.
* ctf-open.c (upgrade_types): Cast to size_t where appropriate.
(ctf_bufopen): Likewise. Use zlib types as needed.
* ctf-types.c (ctf_member_iter): Drop CTF_ERR usage for functions
returning int.
(ctf_enum_iter): Likewise.
(ctf_type_size): Likewise.
(ctf_type_align): Likewise. Cast to size_t where appropriate.
(ctf_type_kind_unsliced): Likewise.
(ctf_type_kind): Likewise.
(ctf_type_encoding): Likewise.
(ctf_member_info): Likewise.
(ctf_array_info): Likewise.
(ctf_enum_value): Likewise.
(ctf_type_rvisit): Likewise.
* ctf-open-bfd.c (ctf_bfdopen): Drop cts_type, cts_flags and
cts_offset.
(ctf_simple_open): Likewise.
(ctf_bfdopen_ctfsect): Likewise. Set cts_size properly.
* Makefile.in: Regenerate.
* aclocal.m4: Likewise.
* config.h: Likewise.
* configure: Likewise.
2019-05-31 17:10:51 +08:00
|
|
|
if (ref == CTF_ERR || ref > CTF_MAX_TYPE)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return (ctf_set_errno (fp, EINVAL));
|
|
|
|
|
2020-06-03 03:04:24 +08:00
|
|
|
if (ref != 0 && ctf_lookup_by_id (&tmp, ref) == NULL)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
if ((type = ctf_add_generic (fp, flag, NULL, kind, &dtd)) == CTF_ERR)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
|
|
|
|
dtd->dtd_data.ctt_info = CTF_TYPE_INFO (kind, flag, 0);
|
|
|
|
dtd->dtd_data.ctt_type = (uint32_t) ref;
|
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
if (kind != CTF_K_POINTER)
|
|
|
|
return type;
|
|
|
|
|
|
|
|
/* If we are adding a pointer, update the ptrtab, both the directly pointed-to
|
|
|
|
type and (if an anonymous typedef node is being pointed at) the type that
|
|
|
|
points at too. Note that ctf_typemax is at this point one higher than we
|
|
|
|
want to check against, because it's just been incremented for the addition
|
|
|
|
of this type. */
|
|
|
|
|
|
|
|
uint32_t type_idx = LCTF_TYPE_TO_INDEX (fp, type);
|
|
|
|
uint32_t ref_idx = LCTF_TYPE_TO_INDEX (fp, ref);
|
|
|
|
|
|
|
|
if (LCTF_TYPE_ISCHILD (fp, ref) == child
|
|
|
|
&& ref_idx < fp->ctf_typemax)
|
|
|
|
{
|
|
|
|
fp->ctf_ptrtab[ref_idx] = type_idx;
|
|
|
|
|
|
|
|
ctf_id_t refref_idx = LCTF_TYPE_TO_INDEX (fp, dtd->dtd_data.ctt_type);
|
|
|
|
|
|
|
|
if (tmp == fp
|
|
|
|
&& (LCTF_INFO_KIND (fp, dtd->dtd_data.ctt_info) == CTF_K_TYPEDEF)
|
|
|
|
&& strcmp (ctf_strptr (fp, dtd->dtd_data.ctt_name), "") == 0
|
|
|
|
&& refref_idx < fp->ctf_typemax)
|
|
|
|
fp->ctf_ptrtab[refref_idx] = type_idx;
|
|
|
|
}
|
|
|
|
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return type;
|
|
|
|
}
|
|
|
|
|
|
|
|
ctf_id_t
|
|
|
|
ctf_add_slice (ctf_file_t *fp, uint32_t flag, ctf_id_t ref,
|
|
|
|
const ctf_encoding_t *ep)
|
|
|
|
{
|
|
|
|
ctf_dtdef_t *dtd;
|
2020-06-03 03:43:03 +08:00
|
|
|
ctf_id_t resolved_ref = ref;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
ctf_id_t type;
|
|
|
|
int kind;
|
|
|
|
const ctf_type_t *tp;
|
|
|
|
ctf_file_t *tmp = fp;
|
|
|
|
|
|
|
|
if (ep == NULL)
|
|
|
|
return (ctf_set_errno (fp, EINVAL));
|
|
|
|
|
|
|
|
if ((ep->cte_bits > 255) || (ep->cte_offset > 255))
|
|
|
|
return (ctf_set_errno (fp, ECTF_SLICEOVERFLOW));
|
|
|
|
|
libctf: fix a number of build problems found on Solaris and NetBSD
- Use of nonportable <endian.h>
- Use of qsort_r
- Use of zlib without appropriate magic to pull in the binutils zlib
- Use of off64_t without checking (fixed by dropping the unused fields
that need off64_t entirely)
- signedness problems due to long being too short a type on 32-bit
platforms: ctf_id_t is now 'unsigned long', and CTF_ERR must be
used only for functions that return ctf_id_t
- One lingering use of bzero() and of <sys/errno.h>
All fixed, using code from gnulib where possible.
Relatedly, set cts_size in a couple of places it was missed
(string table and symbol table loading upon ctf_bfdopen()).
binutils/
* objdump.c (make_ctfsect): Drop cts_type, cts_flags, and
cts_offset.
* readelf.c (shdr_to_ctf_sect): Likewise.
include/
* ctf-api.h (ctf_sect_t): Drop cts_type, cts_flags, and cts_offset.
(ctf_id_t): This is now an unsigned type.
(CTF_ERR): Cast it to ctf_id_t. Note that it should only be used
for ctf_id_t-returning functions.
libctf/
* Makefile.am (ZLIB): New.
(ZLIBINC): Likewise.
(AM_CFLAGS): Use them.
(libctf_a_LIBADD): New, for LIBOBJS.
* configure.ac: Check for zlib, endian.h, and qsort_r.
* ctf-endian.h: New, providing htole64 and le64toh.
* swap.h: Code style fixes.
(bswap_identity_64): New.
* qsort_r.c: New, from gnulib (with one added #include).
* ctf-decls.h: New, providing a conditional qsort_r declaration,
and unconditional definitions of MIN and MAX.
* ctf-impl.h: Use it. Do not use <sys/errno.h>.
(ctf_set_errno): Now returns unsigned long.
* ctf-util.c (ctf_set_errno): Adjust here too.
* ctf-archive.c: Use ctf-endian.h.
(ctf_arc_open_by_offset): Use memset, not bzero. Drop cts_type,
cts_flags and cts_offset.
(ctf_arc_write): Drop debugging dependent on the size of off_t.
* ctf-create.c: Provide a definition of roundup if not defined.
(ctf_create): Drop cts_type, cts_flags and cts_offset.
(ctf_add_reftype): Do not check if type IDs are below zero.
(ctf_add_slice): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_member_offset): Cast error-returning ssize_t's to size_t
when known error-free. Drop CTF_ERR usage for functions returning
int.
(ctf_add_member_encoded): Drop CTF_ERR usage for functions returning
int.
(ctf_add_variable): Likewise.
(enumcmp): Likewise.
(enumadd): Likewise.
(membcmp): Likewise.
(ctf_add_type): Likewise. Cast error-returning ssize_t's to size_t
when known error-free.
* ctf-dump.c (ctf_is_slice): Drop CTF_ERR usage for functions
returning int: use CTF_ERR for functions returning ctf_type_id.
(ctf_dump_label): Likewise.
(ctf_dump_objts): Likewise.
* ctf-labels.c (ctf_label_topmost): Likewise.
(ctf_label_iter): Likewise.
(ctf_label_info): Likewise.
* ctf-lookup.c (ctf_func_args): Likewise.
* ctf-open.c (upgrade_types): Cast to size_t where appropriate.
(ctf_bufopen): Likewise. Use zlib types as needed.
* ctf-types.c (ctf_member_iter): Drop CTF_ERR usage for functions
returning int.
(ctf_enum_iter): Likewise.
(ctf_type_size): Likewise.
(ctf_type_align): Likewise. Cast to size_t where appropriate.
(ctf_type_kind_unsliced): Likewise.
(ctf_type_kind): Likewise.
(ctf_type_encoding): Likewise.
(ctf_member_info): Likewise.
(ctf_array_info): Likewise.
(ctf_enum_value): Likewise.
(ctf_type_rvisit): Likewise.
* ctf-open-bfd.c (ctf_bfdopen): Drop cts_type, cts_flags and
cts_offset.
(ctf_simple_open): Likewise.
(ctf_bfdopen_ctfsect): Likewise. Set cts_size properly.
* Makefile.in: Regenerate.
* aclocal.m4: Likewise.
* config.h: Likewise.
* configure: Likewise.
2019-05-31 17:10:51 +08:00
|
|
|
if (ref == CTF_ERR || ref > CTF_MAX_TYPE)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return (ctf_set_errno (fp, EINVAL));
|
|
|
|
|
2020-06-03 03:04:24 +08:00
|
|
|
if (ref != 0 && ((tp = ctf_lookup_by_id (&tmp, ref)) == NULL))
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
|
2020-06-03 03:43:03 +08:00
|
|
|
/* Make sure we ultimately point to an integral type. We also allow slices to
|
|
|
|
point to the unimplemented type, for now, because the compiler can emit
|
|
|
|
such slices, though they're not very much use. */
|
|
|
|
|
|
|
|
resolved_ref = ctf_type_resolve_unsliced (tmp, ref);
|
|
|
|
kind = ctf_type_kind_unsliced (tmp, resolved_ref);
|
|
|
|
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
if ((kind != CTF_K_INTEGER) && (kind != CTF_K_FLOAT) &&
|
2020-06-03 03:04:24 +08:00
|
|
|
(kind != CTF_K_ENUM)
|
|
|
|
&& (ref != 0))
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return (ctf_set_errno (fp, ECTF_NOTINTFP));
|
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
if ((type = ctf_add_generic (fp, flag, NULL, CTF_K_SLICE, &dtd)) == CTF_ERR)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
|
|
|
|
dtd->dtd_data.ctt_info = CTF_TYPE_INFO (CTF_K_SLICE, flag, 0);
|
Use CHAR_BIT instead of NBBY in libctf
On x86-64 Fedora 29, I tried to build a mingw-hosted gdb that targets
ppc-linux. You can do this with:
../binutils-gdb/configure --host=i686-w64-mingw32 --target=ppc-linux \
--disable-{binutils,gas,gold,gprof,ld}
The build failed with these errors in libctf:
In file included from ../../binutils-gdb/libctf/ctf-create.c:20:
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_encoded':
../../binutils-gdb/libctf/ctf-create.c:803:59: error: 'NBBY' undeclared (first use in this function)
dtd->dtd_data.ctt_size = clp2 (P2ROUNDUP (ep->cte_bits, NBBY) / NBBY);
^~~~
../../binutils-gdb/libctf/ctf-impl.h:254:42: note: in definition of macro 'P2ROUNDUP'
#define P2ROUNDUP(x, align) (-(-(x) & -(align)))
^~~~~
../../binutils-gdb/libctf/ctf-create.c:803:59: note: each undeclared identifier is reported only once for each function it appears in
dtd->dtd_data.ctt_size = clp2 (P2ROUNDUP (ep->cte_bits, NBBY) / NBBY);
^~~~
../../binutils-gdb/libctf/ctf-impl.h:254:42: note: in definition of macro 'P2ROUNDUP'
#define P2ROUNDUP(x, align) (-(-(x) & -(align)))
^~~~~
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_slice':
../../binutils-gdb/libctf/ctf-create.c:862:59: error: 'NBBY' undeclared (first use in this function)
dtd->dtd_data.ctt_size = clp2 (P2ROUNDUP (ep->cte_bits, NBBY) / NBBY);
^~~~
../../binutils-gdb/libctf/ctf-impl.h:254:42: note: in definition of macro 'P2ROUNDUP'
#define P2ROUNDUP(x, align) (-(-(x) & -(align)))
^~~~~
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_member_offset':
../../binutils-gdb/libctf/ctf-create.c:1341:21: error: 'NBBY' undeclared (first use in this function)
off += lsize * NBBY;
^~~~
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_type':
../../binutils-gdb/libctf/ctf-create.c:1822:16: warning: unknown conversion type character 'z' in format [-Wformat=]
ctf_dprintf ("Conflict for type %s against ID %lx: "
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../binutils-gdb/libctf/ctf-create.c:1823:35: note: format string is defined here
"union size differs, old %zi, new %zi\n",
^
../../binutils-gdb/libctf/ctf-create.c:1822:16: warning: unknown conversion type character 'z' in format [-Wformat=]
ctf_dprintf ("Conflict for type %s against ID %lx: "
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../binutils-gdb/libctf/ctf-create.c:1823:44: note: format string is defined here
"union size differs, old %zi, new %zi\n",
^
../../binutils-gdb/libctf/ctf-create.c:1822:16: warning: too many arguments for format [-Wformat-extra-args]
ctf_dprintf ("Conflict for type %s against ID %lx: "
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This patch fixes the actual errors in here. I did not try to fix the
printf warnings, though I think someone ought to.
Ok?
libctf/ChangeLog
2019-06-04 Tom Tromey <tromey@adacore.com>
* ctf-create.c (ctf_add_encoded, ctf_add_slice)
(ctf_add_member_offset): Use CHAR_BIT, not NBBY.
2019-06-05 02:16:57 +08:00
|
|
|
dtd->dtd_data.ctt_size = clp2 (P2ROUNDUP (ep->cte_bits, CHAR_BIT)
|
|
|
|
/ CHAR_BIT);
|
2020-06-03 03:23:07 +08:00
|
|
|
dtd->dtd_u.dtu_slice.cts_type = (uint32_t) ref;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
dtd->dtd_u.dtu_slice.cts_bits = ep->cte_bits;
|
|
|
|
dtd->dtd_u.dtu_slice.cts_offset = ep->cte_offset;
|
|
|
|
|
|
|
|
return type;
|
|
|
|
}
|
|
|
|
|
|
|
|
ctf_id_t
|
|
|
|
ctf_add_integer (ctf_file_t *fp, uint32_t flag,
|
|
|
|
const char *name, const ctf_encoding_t *ep)
|
|
|
|
{
|
|
|
|
return (ctf_add_encoded (fp, flag, name, ep, CTF_K_INTEGER));
|
|
|
|
}
|
|
|
|
|
|
|
|
ctf_id_t
|
|
|
|
ctf_add_float (ctf_file_t *fp, uint32_t flag,
|
|
|
|
const char *name, const ctf_encoding_t *ep)
|
|
|
|
{
|
|
|
|
return (ctf_add_encoded (fp, flag, name, ep, CTF_K_FLOAT));
|
|
|
|
}
|
|
|
|
|
|
|
|
ctf_id_t
|
|
|
|
ctf_add_pointer (ctf_file_t *fp, uint32_t flag, ctf_id_t ref)
|
|
|
|
{
|
|
|
|
return (ctf_add_reftype (fp, flag, ref, CTF_K_POINTER));
|
|
|
|
}
|
|
|
|
|
|
|
|
ctf_id_t
|
|
|
|
ctf_add_array (ctf_file_t *fp, uint32_t flag, const ctf_arinfo_t *arp)
|
|
|
|
{
|
|
|
|
ctf_dtdef_t *dtd;
|
|
|
|
ctf_id_t type;
|
|
|
|
ctf_file_t *tmp = fp;
|
|
|
|
|
|
|
|
if (arp == NULL)
|
|
|
|
return (ctf_set_errno (fp, EINVAL));
|
|
|
|
|
2020-06-03 03:04:24 +08:00
|
|
|
if (arp->ctr_contents != 0
|
|
|
|
&& ctf_lookup_by_id (&tmp, arp->ctr_contents) == NULL)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
|
|
|
|
tmp = fp;
|
|
|
|
if (ctf_lookup_by_id (&tmp, arp->ctr_index) == NULL)
|
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
if ((type = ctf_add_generic (fp, flag, NULL, CTF_K_ARRAY, &dtd)) == CTF_ERR)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
|
|
|
|
dtd->dtd_data.ctt_info = CTF_TYPE_INFO (CTF_K_ARRAY, flag, 0);
|
|
|
|
dtd->dtd_data.ctt_size = 0;
|
|
|
|
dtd->dtd_u.dtu_arr = *arp;
|
|
|
|
|
|
|
|
return type;
|
|
|
|
}
|
|
|
|
|
|
|
|
int
|
|
|
|
ctf_set_array (ctf_file_t *fp, ctf_id_t type, const ctf_arinfo_t *arp)
|
|
|
|
{
|
|
|
|
ctf_dtdef_t *dtd = ctf_dtd_lookup (fp, type);
|
|
|
|
|
|
|
|
if (!(fp->ctf_flags & LCTF_RDWR))
|
|
|
|
return (ctf_set_errno (fp, ECTF_RDONLY));
|
|
|
|
|
|
|
|
if (dtd == NULL
|
|
|
|
|| LCTF_INFO_KIND (fp, dtd->dtd_data.ctt_info) != CTF_K_ARRAY)
|
|
|
|
return (ctf_set_errno (fp, ECTF_BADID));
|
|
|
|
|
|
|
|
fp->ctf_flags |= LCTF_DIRTY;
|
|
|
|
dtd->dtd_u.dtu_arr = *arp;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
ctf_id_t
|
|
|
|
ctf_add_function (ctf_file_t *fp, uint32_t flag,
|
|
|
|
const ctf_funcinfo_t *ctc, const ctf_id_t *argv)
|
|
|
|
{
|
|
|
|
ctf_dtdef_t *dtd;
|
|
|
|
ctf_id_t type;
|
|
|
|
uint32_t vlen;
|
libctf, create: do not corrupt function types' arglists at insertion time
ctf_add_function assumes that function types' arglists are of type
ctf_id_t. Since they are CTF IDs, they are 32 bits wide, a uint32_t:
unfortunately ctf_id_t is a forward-compatible user-facing 64 bits wide,
and should never ever reach the CTF storage level.
All the CTF code other than ctf_add_function correctly assumes that
function arglists outside dynamic containers are 32 bits wide, so the
serialization machinery ends up cutting off half the arglist, corrupting
all args but the first (a good sign is a bunch of args of ID 0, the
unimplemented type, popping up).
Fix this by copying the arglist into place item by item, casting it
properly, at the same time as we validate the arg types. Fix the type
of the dtu_argv in the dynamic container and drop the now-unnecessary
cast in the serializer.
libctf/
* ctf-impl.h (ctf_dtdef_t) <dtu_argv>: Fix type.
* ctf-create.c (ctf_add_function): Check for unimplemented type
and populate at the same time. Populate one-by-one, not via
memcpy.
(ctf_serialize): Remove unnecessary cast.
* ctf-types.c (ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise. Fix comment typo.
2020-06-03 03:17:51 +08:00
|
|
|
uint32_t *vdat = NULL;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
ctf_file_t *tmp = fp;
|
|
|
|
size_t i;
|
|
|
|
|
|
|
|
if (ctc == NULL || (ctc->ctc_flags & ~CTF_FUNC_VARARG) != 0
|
|
|
|
|| (ctc->ctc_argc != 0 && argv == NULL))
|
|
|
|
return (ctf_set_errno (fp, EINVAL));
|
|
|
|
|
|
|
|
vlen = ctc->ctc_argc;
|
|
|
|
if (ctc->ctc_flags & CTF_FUNC_VARARG)
|
|
|
|
vlen++; /* Add trailing zero to indicate varargs (see below). */
|
|
|
|
|
2020-06-03 03:04:24 +08:00
|
|
|
if (ctc->ctc_return != 0
|
|
|
|
&& ctf_lookup_by_id (&tmp, ctc->ctc_return) == NULL)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
|
|
|
|
if (vlen > CTF_MAX_VLEN)
|
|
|
|
return (ctf_set_errno (fp, EOVERFLOW));
|
|
|
|
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
if (vlen != 0 && (vdat = malloc (sizeof (ctf_id_t) * vlen)) == NULL)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return (ctf_set_errno (fp, EAGAIN));
|
|
|
|
|
libctf, create: do not corrupt function types' arglists at insertion time
ctf_add_function assumes that function types' arglists are of type
ctf_id_t. Since they are CTF IDs, they are 32 bits wide, a uint32_t:
unfortunately ctf_id_t is a forward-compatible user-facing 64 bits wide,
and should never ever reach the CTF storage level.
All the CTF code other than ctf_add_function correctly assumes that
function arglists outside dynamic containers are 32 bits wide, so the
serialization machinery ends up cutting off half the arglist, corrupting
all args but the first (a good sign is a bunch of args of ID 0, the
unimplemented type, popping up).
Fix this by copying the arglist into place item by item, casting it
properly, at the same time as we validate the arg types. Fix the type
of the dtu_argv in the dynamic container and drop the now-unnecessary
cast in the serializer.
libctf/
* ctf-impl.h (ctf_dtdef_t) <dtu_argv>: Fix type.
* ctf-create.c (ctf_add_function): Check for unimplemented type
and populate at the same time. Populate one-by-one, not via
memcpy.
(ctf_serialize): Remove unnecessary cast.
* ctf-types.c (ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise. Fix comment typo.
2020-06-03 03:17:51 +08:00
|
|
|
for (i = 0; i < ctc->ctc_argc; i++)
|
|
|
|
{
|
|
|
|
tmp = fp;
|
|
|
|
if (argv[i] != 0 && ctf_lookup_by_id (&tmp, argv[i]) == NULL)
|
|
|
|
{
|
|
|
|
free (vdat);
|
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
}
|
|
|
|
vdat[i] = (uint32_t) argv[i];
|
|
|
|
}
|
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
if ((type = ctf_add_generic (fp, flag, NULL, CTF_K_FUNCTION,
|
|
|
|
&dtd)) == CTF_ERR)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
{
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
free (vdat);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
}
|
|
|
|
|
|
|
|
dtd->dtd_data.ctt_info = CTF_TYPE_INFO (CTF_K_FUNCTION, flag, vlen);
|
|
|
|
dtd->dtd_data.ctt_type = (uint32_t) ctc->ctc_return;
|
|
|
|
|
|
|
|
if (ctc->ctc_flags & CTF_FUNC_VARARG)
|
|
|
|
vdat[vlen - 1] = 0; /* Add trailing zero to indicate varargs. */
|
|
|
|
dtd->dtd_u.dtu_argv = vdat;
|
|
|
|
|
|
|
|
return type;
|
|
|
|
}
|
|
|
|
|
|
|
|
ctf_id_t
|
|
|
|
ctf_add_struct_sized (ctf_file_t *fp, uint32_t flag, const char *name,
|
|
|
|
size_t size)
|
|
|
|
{
|
|
|
|
ctf_dtdef_t *dtd;
|
|
|
|
ctf_id_t type = 0;
|
|
|
|
|
2019-10-21 18:27:43 +08:00
|
|
|
/* Promote root-visible forwards to structs. */
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
if (name != NULL)
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
type = ctf_lookup_by_rawname (fp, CTF_K_STRUCT, name);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
|
|
|
if (type != 0 && ctf_type_kind (fp, type) == CTF_K_FORWARD)
|
|
|
|
dtd = ctf_dtd_lookup (fp, type);
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
else if ((type = ctf_add_generic (fp, flag, name, CTF_K_STRUCT,
|
|
|
|
&dtd)) == CTF_ERR)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
|
|
|
|
dtd->dtd_data.ctt_info = CTF_TYPE_INFO (CTF_K_STRUCT, flag, 0);
|
|
|
|
|
|
|
|
if (size > CTF_MAX_SIZE)
|
|
|
|
{
|
|
|
|
dtd->dtd_data.ctt_size = CTF_LSIZE_SENT;
|
|
|
|
dtd->dtd_data.ctt_lsizehi = CTF_SIZE_TO_LSIZE_HI (size);
|
|
|
|
dtd->dtd_data.ctt_lsizelo = CTF_SIZE_TO_LSIZE_LO (size);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
dtd->dtd_data.ctt_size = (uint32_t) size;
|
|
|
|
|
|
|
|
return type;
|
|
|
|
}
|
|
|
|
|
|
|
|
ctf_id_t
|
|
|
|
ctf_add_struct (ctf_file_t *fp, uint32_t flag, const char *name)
|
|
|
|
{
|
|
|
|
return (ctf_add_struct_sized (fp, flag, name, 0));
|
|
|
|
}
|
|
|
|
|
|
|
|
ctf_id_t
|
|
|
|
ctf_add_union_sized (ctf_file_t *fp, uint32_t flag, const char *name,
|
|
|
|
size_t size)
|
|
|
|
{
|
|
|
|
ctf_dtdef_t *dtd;
|
|
|
|
ctf_id_t type = 0;
|
|
|
|
|
2019-10-21 18:27:43 +08:00
|
|
|
/* Promote root-visible forwards to unions. */
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
if (name != NULL)
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
type = ctf_lookup_by_rawname (fp, CTF_K_UNION, name);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
|
|
|
if (type != 0 && ctf_type_kind (fp, type) == CTF_K_FORWARD)
|
|
|
|
dtd = ctf_dtd_lookup (fp, type);
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
else if ((type = ctf_add_generic (fp, flag, name, CTF_K_UNION,
|
|
|
|
&dtd)) == CTF_ERR)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return CTF_ERR; /* errno is set for us */
|
|
|
|
|
|
|
|
dtd->dtd_data.ctt_info = CTF_TYPE_INFO (CTF_K_UNION, flag, 0);
|
|
|
|
|
|
|
|
if (size > CTF_MAX_SIZE)
|
|
|
|
{
|
|
|
|
dtd->dtd_data.ctt_size = CTF_LSIZE_SENT;
|
|
|
|
dtd->dtd_data.ctt_lsizehi = CTF_SIZE_TO_LSIZE_HI (size);
|
|
|
|
dtd->dtd_data.ctt_lsizelo = CTF_SIZE_TO_LSIZE_LO (size);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
dtd->dtd_data.ctt_size = (uint32_t) size;
|
|
|
|
|
|
|
|
return type;
|
|
|
|
}
|
|
|
|
|
|
|
|
ctf_id_t
|
|
|
|
ctf_add_union (ctf_file_t *fp, uint32_t flag, const char *name)
|
|
|
|
{
|
|
|
|
return (ctf_add_union_sized (fp, flag, name, 0));
|
|
|
|
}
|
|
|
|
|
|
|
|
ctf_id_t
|
|
|
|
ctf_add_enum (ctf_file_t *fp, uint32_t flag, const char *name)
|
|
|
|
{
|
|
|
|
ctf_dtdef_t *dtd;
|
|
|
|
ctf_id_t type = 0;
|
|
|
|
|
2019-10-21 18:27:43 +08:00
|
|
|
/* Promote root-visible forwards to enums. */
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
if (name != NULL)
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
type = ctf_lookup_by_rawname (fp, CTF_K_ENUM, name);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
|
|
|
if (type != 0 && ctf_type_kind (fp, type) == CTF_K_FORWARD)
|
|
|
|
dtd = ctf_dtd_lookup (fp, type);
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
else if ((type = ctf_add_generic (fp, flag, name, CTF_K_ENUM,
|
|
|
|
&dtd)) == CTF_ERR)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
|
|
|
|
dtd->dtd_data.ctt_info = CTF_TYPE_INFO (CTF_K_ENUM, flag, 0);
|
|
|
|
dtd->dtd_data.ctt_size = fp->ctf_dmodel->ctd_int;
|
|
|
|
|
|
|
|
return type;
|
|
|
|
}
|
|
|
|
|
|
|
|
ctf_id_t
|
|
|
|
ctf_add_enum_encoded (ctf_file_t *fp, uint32_t flag, const char *name,
|
|
|
|
const ctf_encoding_t *ep)
|
|
|
|
{
|
|
|
|
ctf_id_t type = 0;
|
|
|
|
|
|
|
|
/* First, create the enum if need be, using most of the same machinery as
|
|
|
|
ctf_add_enum(), to ensure that we do not allow things past that are not
|
|
|
|
enums or forwards to them. (This includes other slices: you cannot slice a
|
|
|
|
slice, which would be a useless thing to do anyway.) */
|
|
|
|
|
|
|
|
if (name != NULL)
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
type = ctf_lookup_by_rawname (fp, CTF_K_ENUM, name);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
|
|
|
if (type != 0)
|
|
|
|
{
|
|
|
|
if ((ctf_type_kind (fp, type) != CTF_K_FORWARD) &&
|
|
|
|
(ctf_type_kind_unsliced (fp, type) != CTF_K_ENUM))
|
|
|
|
return (ctf_set_errno (fp, ECTF_NOTINTFP));
|
|
|
|
}
|
|
|
|
else if ((type = ctf_add_enum (fp, flag, name)) == CTF_ERR)
|
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
|
|
|
|
/* Now attach a suitable slice to it. */
|
|
|
|
|
|
|
|
return ctf_add_slice (fp, flag, type, ep);
|
|
|
|
}
|
|
|
|
|
|
|
|
ctf_id_t
|
|
|
|
ctf_add_forward (ctf_file_t *fp, uint32_t flag, const char *name,
|
|
|
|
uint32_t kind)
|
|
|
|
{
|
|
|
|
ctf_dtdef_t *dtd;
|
|
|
|
ctf_id_t type = 0;
|
|
|
|
|
2020-06-03 20:23:42 +08:00
|
|
|
if (!ctf_forwardable_kind (kind))
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
return (ctf_set_errno (fp, ECTF_NOTSUE));
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
|
|
|
/* If the type is already defined or exists as a forward tag, just
|
|
|
|
return the ctf_id_t of the existing definition. */
|
|
|
|
|
|
|
|
if (name != NULL)
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
type = ctf_lookup_by_rawname (fp, kind, name);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
2019-10-21 18:33:19 +08:00
|
|
|
if (type)
|
|
|
|
return type;
|
|
|
|
|
libctf: create: forwards are always in the namespace of their referent
The C namespace a forward is located in is always the same as the
namespace of the corresponding complete type: 'struct foo' is in the
struct namespace and does not collide with, say, 'union foo'.
libctf allowed for this in many places, but inconsistently: in
particular, forward *addition* never allowed for this, and was interning
forwards in the default namespace, which is always wrong, since you can
only forward structs, unions and enums, all of which are in their own
namespaces in C.
Forward removal needs corresponding adjustment to remove the names form
the right namespace, as does ctf_rollback.
libctf/
* ctf-create.c (ctf_add_forward): Intern in the right namespace.
(ctf_dtd_delete): Remove correspondingly.
(ctf_rollback): Likewise.
2019-11-06 01:57:55 +08:00
|
|
|
if ((type = ctf_add_generic (fp, flag, name, kind, &dtd)) == CTF_ERR)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
|
|
|
|
dtd->dtd_data.ctt_info = CTF_TYPE_INFO (CTF_K_FORWARD, flag, 0);
|
|
|
|
dtd->dtd_data.ctt_type = kind;
|
|
|
|
|
|
|
|
return type;
|
|
|
|
}
|
|
|
|
|
|
|
|
ctf_id_t
|
|
|
|
ctf_add_typedef (ctf_file_t *fp, uint32_t flag, const char *name,
|
|
|
|
ctf_id_t ref)
|
|
|
|
{
|
|
|
|
ctf_dtdef_t *dtd;
|
|
|
|
ctf_id_t type;
|
|
|
|
ctf_file_t *tmp = fp;
|
|
|
|
|
libctf: fix a number of build problems found on Solaris and NetBSD
- Use of nonportable <endian.h>
- Use of qsort_r
- Use of zlib without appropriate magic to pull in the binutils zlib
- Use of off64_t without checking (fixed by dropping the unused fields
that need off64_t entirely)
- signedness problems due to long being too short a type on 32-bit
platforms: ctf_id_t is now 'unsigned long', and CTF_ERR must be
used only for functions that return ctf_id_t
- One lingering use of bzero() and of <sys/errno.h>
All fixed, using code from gnulib where possible.
Relatedly, set cts_size in a couple of places it was missed
(string table and symbol table loading upon ctf_bfdopen()).
binutils/
* objdump.c (make_ctfsect): Drop cts_type, cts_flags, and
cts_offset.
* readelf.c (shdr_to_ctf_sect): Likewise.
include/
* ctf-api.h (ctf_sect_t): Drop cts_type, cts_flags, and cts_offset.
(ctf_id_t): This is now an unsigned type.
(CTF_ERR): Cast it to ctf_id_t. Note that it should only be used
for ctf_id_t-returning functions.
libctf/
* Makefile.am (ZLIB): New.
(ZLIBINC): Likewise.
(AM_CFLAGS): Use them.
(libctf_a_LIBADD): New, for LIBOBJS.
* configure.ac: Check for zlib, endian.h, and qsort_r.
* ctf-endian.h: New, providing htole64 and le64toh.
* swap.h: Code style fixes.
(bswap_identity_64): New.
* qsort_r.c: New, from gnulib (with one added #include).
* ctf-decls.h: New, providing a conditional qsort_r declaration,
and unconditional definitions of MIN and MAX.
* ctf-impl.h: Use it. Do not use <sys/errno.h>.
(ctf_set_errno): Now returns unsigned long.
* ctf-util.c (ctf_set_errno): Adjust here too.
* ctf-archive.c: Use ctf-endian.h.
(ctf_arc_open_by_offset): Use memset, not bzero. Drop cts_type,
cts_flags and cts_offset.
(ctf_arc_write): Drop debugging dependent on the size of off_t.
* ctf-create.c: Provide a definition of roundup if not defined.
(ctf_create): Drop cts_type, cts_flags and cts_offset.
(ctf_add_reftype): Do not check if type IDs are below zero.
(ctf_add_slice): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_member_offset): Cast error-returning ssize_t's to size_t
when known error-free. Drop CTF_ERR usage for functions returning
int.
(ctf_add_member_encoded): Drop CTF_ERR usage for functions returning
int.
(ctf_add_variable): Likewise.
(enumcmp): Likewise.
(enumadd): Likewise.
(membcmp): Likewise.
(ctf_add_type): Likewise. Cast error-returning ssize_t's to size_t
when known error-free.
* ctf-dump.c (ctf_is_slice): Drop CTF_ERR usage for functions
returning int: use CTF_ERR for functions returning ctf_type_id.
(ctf_dump_label): Likewise.
(ctf_dump_objts): Likewise.
* ctf-labels.c (ctf_label_topmost): Likewise.
(ctf_label_iter): Likewise.
(ctf_label_info): Likewise.
* ctf-lookup.c (ctf_func_args): Likewise.
* ctf-open.c (upgrade_types): Cast to size_t where appropriate.
(ctf_bufopen): Likewise. Use zlib types as needed.
* ctf-types.c (ctf_member_iter): Drop CTF_ERR usage for functions
returning int.
(ctf_enum_iter): Likewise.
(ctf_type_size): Likewise.
(ctf_type_align): Likewise. Cast to size_t where appropriate.
(ctf_type_kind_unsliced): Likewise.
(ctf_type_kind): Likewise.
(ctf_type_encoding): Likewise.
(ctf_member_info): Likewise.
(ctf_array_info): Likewise.
(ctf_enum_value): Likewise.
(ctf_type_rvisit): Likewise.
* ctf-open-bfd.c (ctf_bfdopen): Drop cts_type, cts_flags and
cts_offset.
(ctf_simple_open): Likewise.
(ctf_bfdopen_ctfsect): Likewise. Set cts_size properly.
* Makefile.in: Regenerate.
* aclocal.m4: Likewise.
* config.h: Likewise.
* configure: Likewise.
2019-05-31 17:10:51 +08:00
|
|
|
if (ref == CTF_ERR || ref > CTF_MAX_TYPE)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return (ctf_set_errno (fp, EINVAL));
|
|
|
|
|
2020-06-03 03:04:24 +08:00
|
|
|
if (ref != 0 && ctf_lookup_by_id (&tmp, ref) == NULL)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
if ((type = ctf_add_generic (fp, flag, name, CTF_K_TYPEDEF,
|
|
|
|
&dtd)) == CTF_ERR)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
|
|
|
|
dtd->dtd_data.ctt_info = CTF_TYPE_INFO (CTF_K_TYPEDEF, flag, 0);
|
|
|
|
dtd->dtd_data.ctt_type = (uint32_t) ref;
|
|
|
|
|
|
|
|
return type;
|
|
|
|
}
|
|
|
|
|
|
|
|
ctf_id_t
|
|
|
|
ctf_add_volatile (ctf_file_t *fp, uint32_t flag, ctf_id_t ref)
|
|
|
|
{
|
|
|
|
return (ctf_add_reftype (fp, flag, ref, CTF_K_VOLATILE));
|
|
|
|
}
|
|
|
|
|
|
|
|
ctf_id_t
|
|
|
|
ctf_add_const (ctf_file_t *fp, uint32_t flag, ctf_id_t ref)
|
|
|
|
{
|
|
|
|
return (ctf_add_reftype (fp, flag, ref, CTF_K_CONST));
|
|
|
|
}
|
|
|
|
|
|
|
|
ctf_id_t
|
|
|
|
ctf_add_restrict (ctf_file_t *fp, uint32_t flag, ctf_id_t ref)
|
|
|
|
{
|
|
|
|
return (ctf_add_reftype (fp, flag, ref, CTF_K_RESTRICT));
|
|
|
|
}
|
|
|
|
|
|
|
|
int
|
|
|
|
ctf_add_enumerator (ctf_file_t *fp, ctf_id_t enid, const char *name,
|
|
|
|
int value)
|
|
|
|
{
|
|
|
|
ctf_dtdef_t *dtd = ctf_dtd_lookup (fp, enid);
|
|
|
|
ctf_dmdef_t *dmd;
|
|
|
|
|
|
|
|
uint32_t kind, vlen, root;
|
|
|
|
char *s;
|
|
|
|
|
|
|
|
if (name == NULL)
|
|
|
|
return (ctf_set_errno (fp, EINVAL));
|
|
|
|
|
|
|
|
if (!(fp->ctf_flags & LCTF_RDWR))
|
|
|
|
return (ctf_set_errno (fp, ECTF_RDONLY));
|
|
|
|
|
|
|
|
if (dtd == NULL)
|
|
|
|
return (ctf_set_errno (fp, ECTF_BADID));
|
|
|
|
|
|
|
|
kind = LCTF_INFO_KIND (fp, dtd->dtd_data.ctt_info);
|
|
|
|
root = LCTF_INFO_ISROOT (fp, dtd->dtd_data.ctt_info);
|
|
|
|
vlen = LCTF_INFO_VLEN (fp, dtd->dtd_data.ctt_info);
|
|
|
|
|
|
|
|
if (kind != CTF_K_ENUM)
|
|
|
|
return (ctf_set_errno (fp, ECTF_NOTENUM));
|
|
|
|
|
|
|
|
if (vlen == CTF_MAX_VLEN)
|
|
|
|
return (ctf_set_errno (fp, ECTF_DTFULL));
|
|
|
|
|
|
|
|
for (dmd = ctf_list_next (&dtd->dtd_u.dtu_members);
|
|
|
|
dmd != NULL; dmd = ctf_list_next (dmd))
|
|
|
|
{
|
|
|
|
if (strcmp (dmd->dmd_name, name) == 0)
|
|
|
|
return (ctf_set_errno (fp, ECTF_DUPLICATE));
|
|
|
|
}
|
|
|
|
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
if ((dmd = malloc (sizeof (ctf_dmdef_t))) == NULL)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return (ctf_set_errno (fp, EAGAIN));
|
|
|
|
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
if ((s = strdup (name)) == NULL)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
{
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
free (dmd);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return (ctf_set_errno (fp, EAGAIN));
|
|
|
|
}
|
|
|
|
|
|
|
|
dmd->dmd_name = s;
|
|
|
|
dmd->dmd_type = CTF_ERR;
|
|
|
|
dmd->dmd_offset = 0;
|
|
|
|
dmd->dmd_value = value;
|
|
|
|
|
|
|
|
dtd->dtd_data.ctt_info = CTF_TYPE_INFO (kind, root, vlen + 1);
|
|
|
|
ctf_list_append (&dtd->dtd_u.dtu_members, dmd);
|
|
|
|
|
|
|
|
fp->ctf_flags |= LCTF_DIRTY;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
int
|
|
|
|
ctf_add_member_offset (ctf_file_t *fp, ctf_id_t souid, const char *name,
|
|
|
|
ctf_id_t type, unsigned long bit_offset)
|
|
|
|
{
|
|
|
|
ctf_dtdef_t *dtd = ctf_dtd_lookup (fp, souid);
|
|
|
|
ctf_dmdef_t *dmd;
|
|
|
|
|
|
|
|
ssize_t msize, malign, ssize;
|
|
|
|
uint32_t kind, vlen, root;
|
|
|
|
char *s = NULL;
|
|
|
|
|
|
|
|
if (!(fp->ctf_flags & LCTF_RDWR))
|
|
|
|
return (ctf_set_errno (fp, ECTF_RDONLY));
|
|
|
|
|
|
|
|
if (dtd == NULL)
|
|
|
|
return (ctf_set_errno (fp, ECTF_BADID));
|
|
|
|
|
2020-06-03 03:28:16 +08:00
|
|
|
if (name != NULL && name[0] == '\0')
|
|
|
|
name = NULL;
|
|
|
|
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
kind = LCTF_INFO_KIND (fp, dtd->dtd_data.ctt_info);
|
|
|
|
root = LCTF_INFO_ISROOT (fp, dtd->dtd_data.ctt_info);
|
|
|
|
vlen = LCTF_INFO_VLEN (fp, dtd->dtd_data.ctt_info);
|
|
|
|
|
|
|
|
if (kind != CTF_K_STRUCT && kind != CTF_K_UNION)
|
|
|
|
return (ctf_set_errno (fp, ECTF_NOTSOU));
|
|
|
|
|
|
|
|
if (vlen == CTF_MAX_VLEN)
|
|
|
|
return (ctf_set_errno (fp, ECTF_DTFULL));
|
|
|
|
|
|
|
|
if (name != NULL)
|
|
|
|
{
|
|
|
|
for (dmd = ctf_list_next (&dtd->dtd_u.dtu_members);
|
|
|
|
dmd != NULL; dmd = ctf_list_next (dmd))
|
|
|
|
{
|
|
|
|
if (dmd->dmd_name != NULL && strcmp (dmd->dmd_name, name) == 0)
|
|
|
|
return (ctf_set_errno (fp, ECTF_DUPLICATE));
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
libctf: fix a number of build problems found on Solaris and NetBSD
- Use of nonportable <endian.h>
- Use of qsort_r
- Use of zlib without appropriate magic to pull in the binutils zlib
- Use of off64_t without checking (fixed by dropping the unused fields
that need off64_t entirely)
- signedness problems due to long being too short a type on 32-bit
platforms: ctf_id_t is now 'unsigned long', and CTF_ERR must be
used only for functions that return ctf_id_t
- One lingering use of bzero() and of <sys/errno.h>
All fixed, using code from gnulib where possible.
Relatedly, set cts_size in a couple of places it was missed
(string table and symbol table loading upon ctf_bfdopen()).
binutils/
* objdump.c (make_ctfsect): Drop cts_type, cts_flags, and
cts_offset.
* readelf.c (shdr_to_ctf_sect): Likewise.
include/
* ctf-api.h (ctf_sect_t): Drop cts_type, cts_flags, and cts_offset.
(ctf_id_t): This is now an unsigned type.
(CTF_ERR): Cast it to ctf_id_t. Note that it should only be used
for ctf_id_t-returning functions.
libctf/
* Makefile.am (ZLIB): New.
(ZLIBINC): Likewise.
(AM_CFLAGS): Use them.
(libctf_a_LIBADD): New, for LIBOBJS.
* configure.ac: Check for zlib, endian.h, and qsort_r.
* ctf-endian.h: New, providing htole64 and le64toh.
* swap.h: Code style fixes.
(bswap_identity_64): New.
* qsort_r.c: New, from gnulib (with one added #include).
* ctf-decls.h: New, providing a conditional qsort_r declaration,
and unconditional definitions of MIN and MAX.
* ctf-impl.h: Use it. Do not use <sys/errno.h>.
(ctf_set_errno): Now returns unsigned long.
* ctf-util.c (ctf_set_errno): Adjust here too.
* ctf-archive.c: Use ctf-endian.h.
(ctf_arc_open_by_offset): Use memset, not bzero. Drop cts_type,
cts_flags and cts_offset.
(ctf_arc_write): Drop debugging dependent on the size of off_t.
* ctf-create.c: Provide a definition of roundup if not defined.
(ctf_create): Drop cts_type, cts_flags and cts_offset.
(ctf_add_reftype): Do not check if type IDs are below zero.
(ctf_add_slice): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_member_offset): Cast error-returning ssize_t's to size_t
when known error-free. Drop CTF_ERR usage for functions returning
int.
(ctf_add_member_encoded): Drop CTF_ERR usage for functions returning
int.
(ctf_add_variable): Likewise.
(enumcmp): Likewise.
(enumadd): Likewise.
(membcmp): Likewise.
(ctf_add_type): Likewise. Cast error-returning ssize_t's to size_t
when known error-free.
* ctf-dump.c (ctf_is_slice): Drop CTF_ERR usage for functions
returning int: use CTF_ERR for functions returning ctf_type_id.
(ctf_dump_label): Likewise.
(ctf_dump_objts): Likewise.
* ctf-labels.c (ctf_label_topmost): Likewise.
(ctf_label_iter): Likewise.
(ctf_label_info): Likewise.
* ctf-lookup.c (ctf_func_args): Likewise.
* ctf-open.c (upgrade_types): Cast to size_t where appropriate.
(ctf_bufopen): Likewise. Use zlib types as needed.
* ctf-types.c (ctf_member_iter): Drop CTF_ERR usage for functions
returning int.
(ctf_enum_iter): Likewise.
(ctf_type_size): Likewise.
(ctf_type_align): Likewise. Cast to size_t where appropriate.
(ctf_type_kind_unsliced): Likewise.
(ctf_type_kind): Likewise.
(ctf_type_encoding): Likewise.
(ctf_member_info): Likewise.
(ctf_array_info): Likewise.
(ctf_enum_value): Likewise.
(ctf_type_rvisit): Likewise.
* ctf-open-bfd.c (ctf_bfdopen): Drop cts_type, cts_flags and
cts_offset.
(ctf_simple_open): Likewise.
(ctf_bfdopen_ctfsect): Likewise. Set cts_size properly.
* Makefile.in: Regenerate.
* aclocal.m4: Likewise.
* config.h: Likewise.
* configure: Likewise.
2019-05-31 17:10:51 +08:00
|
|
|
if ((msize = ctf_type_size (fp, type)) < 0 ||
|
|
|
|
(malign = ctf_type_align (fp, type)) < 0)
|
2020-06-03 03:04:24 +08:00
|
|
|
{
|
|
|
|
/* The unimplemented type, and any type that resolves to it, has no size
|
|
|
|
and no alignment: it can correspond to any number of compiler-inserted
|
|
|
|
types. */
|
|
|
|
|
|
|
|
if (ctf_errno (fp) == ECTF_NONREPRESENTABLE)
|
|
|
|
{
|
|
|
|
msize = 0;
|
|
|
|
malign = 0;
|
|
|
|
ctf_set_errno (fp, 0);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
return -1; /* errno is set for us. */
|
|
|
|
}
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
if ((dmd = malloc (sizeof (ctf_dmdef_t))) == NULL)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return (ctf_set_errno (fp, EAGAIN));
|
|
|
|
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
if (name != NULL && (s = strdup (name)) == NULL)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
{
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
free (dmd);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return (ctf_set_errno (fp, EAGAIN));
|
|
|
|
}
|
|
|
|
|
|
|
|
dmd->dmd_name = s;
|
|
|
|
dmd->dmd_type = type;
|
|
|
|
dmd->dmd_value = -1;
|
|
|
|
|
|
|
|
if (kind == CTF_K_STRUCT && vlen != 0)
|
|
|
|
{
|
|
|
|
if (bit_offset == (unsigned long) - 1)
|
|
|
|
{
|
|
|
|
/* Natural alignment. */
|
|
|
|
|
|
|
|
ctf_dmdef_t *lmd = ctf_list_prev (&dtd->dtd_u.dtu_members);
|
|
|
|
ctf_id_t ltype = ctf_type_resolve (fp, lmd->dmd_type);
|
|
|
|
size_t off = lmd->dmd_offset;
|
|
|
|
|
|
|
|
ctf_encoding_t linfo;
|
|
|
|
ssize_t lsize;
|
|
|
|
|
2020-06-03 03:04:24 +08:00
|
|
|
/* Propagate any error from ctf_type_resolve. If the last member was
|
|
|
|
of unimplemented type, this may be -ECTF_NONREPRESENTABLE: we
|
|
|
|
cannot insert right after such a member without explicit offset
|
|
|
|
specification, because its alignment and size is not known. */
|
|
|
|
if (ltype == CTF_ERR)
|
|
|
|
{
|
|
|
|
free (dmd);
|
|
|
|
return -1; /* errno is set for us. */
|
|
|
|
}
|
|
|
|
|
libctf: fix a number of build problems found on Solaris and NetBSD
- Use of nonportable <endian.h>
- Use of qsort_r
- Use of zlib without appropriate magic to pull in the binutils zlib
- Use of off64_t without checking (fixed by dropping the unused fields
that need off64_t entirely)
- signedness problems due to long being too short a type on 32-bit
platforms: ctf_id_t is now 'unsigned long', and CTF_ERR must be
used only for functions that return ctf_id_t
- One lingering use of bzero() and of <sys/errno.h>
All fixed, using code from gnulib where possible.
Relatedly, set cts_size in a couple of places it was missed
(string table and symbol table loading upon ctf_bfdopen()).
binutils/
* objdump.c (make_ctfsect): Drop cts_type, cts_flags, and
cts_offset.
* readelf.c (shdr_to_ctf_sect): Likewise.
include/
* ctf-api.h (ctf_sect_t): Drop cts_type, cts_flags, and cts_offset.
(ctf_id_t): This is now an unsigned type.
(CTF_ERR): Cast it to ctf_id_t. Note that it should only be used
for ctf_id_t-returning functions.
libctf/
* Makefile.am (ZLIB): New.
(ZLIBINC): Likewise.
(AM_CFLAGS): Use them.
(libctf_a_LIBADD): New, for LIBOBJS.
* configure.ac: Check for zlib, endian.h, and qsort_r.
* ctf-endian.h: New, providing htole64 and le64toh.
* swap.h: Code style fixes.
(bswap_identity_64): New.
* qsort_r.c: New, from gnulib (with one added #include).
* ctf-decls.h: New, providing a conditional qsort_r declaration,
and unconditional definitions of MIN and MAX.
* ctf-impl.h: Use it. Do not use <sys/errno.h>.
(ctf_set_errno): Now returns unsigned long.
* ctf-util.c (ctf_set_errno): Adjust here too.
* ctf-archive.c: Use ctf-endian.h.
(ctf_arc_open_by_offset): Use memset, not bzero. Drop cts_type,
cts_flags and cts_offset.
(ctf_arc_write): Drop debugging dependent on the size of off_t.
* ctf-create.c: Provide a definition of roundup if not defined.
(ctf_create): Drop cts_type, cts_flags and cts_offset.
(ctf_add_reftype): Do not check if type IDs are below zero.
(ctf_add_slice): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_member_offset): Cast error-returning ssize_t's to size_t
when known error-free. Drop CTF_ERR usage for functions returning
int.
(ctf_add_member_encoded): Drop CTF_ERR usage for functions returning
int.
(ctf_add_variable): Likewise.
(enumcmp): Likewise.
(enumadd): Likewise.
(membcmp): Likewise.
(ctf_add_type): Likewise. Cast error-returning ssize_t's to size_t
when known error-free.
* ctf-dump.c (ctf_is_slice): Drop CTF_ERR usage for functions
returning int: use CTF_ERR for functions returning ctf_type_id.
(ctf_dump_label): Likewise.
(ctf_dump_objts): Likewise.
* ctf-labels.c (ctf_label_topmost): Likewise.
(ctf_label_iter): Likewise.
(ctf_label_info): Likewise.
* ctf-lookup.c (ctf_func_args): Likewise.
* ctf-open.c (upgrade_types): Cast to size_t where appropriate.
(ctf_bufopen): Likewise. Use zlib types as needed.
* ctf-types.c (ctf_member_iter): Drop CTF_ERR usage for functions
returning int.
(ctf_enum_iter): Likewise.
(ctf_type_size): Likewise.
(ctf_type_align): Likewise. Cast to size_t where appropriate.
(ctf_type_kind_unsliced): Likewise.
(ctf_type_kind): Likewise.
(ctf_type_encoding): Likewise.
(ctf_member_info): Likewise.
(ctf_array_info): Likewise.
(ctf_enum_value): Likewise.
(ctf_type_rvisit): Likewise.
* ctf-open-bfd.c (ctf_bfdopen): Drop cts_type, cts_flags and
cts_offset.
(ctf_simple_open): Likewise.
(ctf_bfdopen_ctfsect): Likewise. Set cts_size properly.
* Makefile.in: Regenerate.
* aclocal.m4: Likewise.
* config.h: Likewise.
* configure: Likewise.
2019-05-31 17:10:51 +08:00
|
|
|
if (ctf_type_encoding (fp, ltype, &linfo) == 0)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
off += linfo.cte_bits;
|
libctf: fix a number of build problems found on Solaris and NetBSD
- Use of nonportable <endian.h>
- Use of qsort_r
- Use of zlib without appropriate magic to pull in the binutils zlib
- Use of off64_t without checking (fixed by dropping the unused fields
that need off64_t entirely)
- signedness problems due to long being too short a type on 32-bit
platforms: ctf_id_t is now 'unsigned long', and CTF_ERR must be
used only for functions that return ctf_id_t
- One lingering use of bzero() and of <sys/errno.h>
All fixed, using code from gnulib where possible.
Relatedly, set cts_size in a couple of places it was missed
(string table and symbol table loading upon ctf_bfdopen()).
binutils/
* objdump.c (make_ctfsect): Drop cts_type, cts_flags, and
cts_offset.
* readelf.c (shdr_to_ctf_sect): Likewise.
include/
* ctf-api.h (ctf_sect_t): Drop cts_type, cts_flags, and cts_offset.
(ctf_id_t): This is now an unsigned type.
(CTF_ERR): Cast it to ctf_id_t. Note that it should only be used
for ctf_id_t-returning functions.
libctf/
* Makefile.am (ZLIB): New.
(ZLIBINC): Likewise.
(AM_CFLAGS): Use them.
(libctf_a_LIBADD): New, for LIBOBJS.
* configure.ac: Check for zlib, endian.h, and qsort_r.
* ctf-endian.h: New, providing htole64 and le64toh.
* swap.h: Code style fixes.
(bswap_identity_64): New.
* qsort_r.c: New, from gnulib (with one added #include).
* ctf-decls.h: New, providing a conditional qsort_r declaration,
and unconditional definitions of MIN and MAX.
* ctf-impl.h: Use it. Do not use <sys/errno.h>.
(ctf_set_errno): Now returns unsigned long.
* ctf-util.c (ctf_set_errno): Adjust here too.
* ctf-archive.c: Use ctf-endian.h.
(ctf_arc_open_by_offset): Use memset, not bzero. Drop cts_type,
cts_flags and cts_offset.
(ctf_arc_write): Drop debugging dependent on the size of off_t.
* ctf-create.c: Provide a definition of roundup if not defined.
(ctf_create): Drop cts_type, cts_flags and cts_offset.
(ctf_add_reftype): Do not check if type IDs are below zero.
(ctf_add_slice): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_member_offset): Cast error-returning ssize_t's to size_t
when known error-free. Drop CTF_ERR usage for functions returning
int.
(ctf_add_member_encoded): Drop CTF_ERR usage for functions returning
int.
(ctf_add_variable): Likewise.
(enumcmp): Likewise.
(enumadd): Likewise.
(membcmp): Likewise.
(ctf_add_type): Likewise. Cast error-returning ssize_t's to size_t
when known error-free.
* ctf-dump.c (ctf_is_slice): Drop CTF_ERR usage for functions
returning int: use CTF_ERR for functions returning ctf_type_id.
(ctf_dump_label): Likewise.
(ctf_dump_objts): Likewise.
* ctf-labels.c (ctf_label_topmost): Likewise.
(ctf_label_iter): Likewise.
(ctf_label_info): Likewise.
* ctf-lookup.c (ctf_func_args): Likewise.
* ctf-open.c (upgrade_types): Cast to size_t where appropriate.
(ctf_bufopen): Likewise. Use zlib types as needed.
* ctf-types.c (ctf_member_iter): Drop CTF_ERR usage for functions
returning int.
(ctf_enum_iter): Likewise.
(ctf_type_size): Likewise.
(ctf_type_align): Likewise. Cast to size_t where appropriate.
(ctf_type_kind_unsliced): Likewise.
(ctf_type_kind): Likewise.
(ctf_type_encoding): Likewise.
(ctf_member_info): Likewise.
(ctf_array_info): Likewise.
(ctf_enum_value): Likewise.
(ctf_type_rvisit): Likewise.
* ctf-open-bfd.c (ctf_bfdopen): Drop cts_type, cts_flags and
cts_offset.
(ctf_simple_open): Likewise.
(ctf_bfdopen_ctfsect): Likewise. Set cts_size properly.
* Makefile.in: Regenerate.
* aclocal.m4: Likewise.
* config.h: Likewise.
* configure: Likewise.
2019-05-31 17:10:51 +08:00
|
|
|
else if ((lsize = ctf_type_size (fp, ltype)) > 0)
|
Use CHAR_BIT instead of NBBY in libctf
On x86-64 Fedora 29, I tried to build a mingw-hosted gdb that targets
ppc-linux. You can do this with:
../binutils-gdb/configure --host=i686-w64-mingw32 --target=ppc-linux \
--disable-{binutils,gas,gold,gprof,ld}
The build failed with these errors in libctf:
In file included from ../../binutils-gdb/libctf/ctf-create.c:20:
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_encoded':
../../binutils-gdb/libctf/ctf-create.c:803:59: error: 'NBBY' undeclared (first use in this function)
dtd->dtd_data.ctt_size = clp2 (P2ROUNDUP (ep->cte_bits, NBBY) / NBBY);
^~~~
../../binutils-gdb/libctf/ctf-impl.h:254:42: note: in definition of macro 'P2ROUNDUP'
#define P2ROUNDUP(x, align) (-(-(x) & -(align)))
^~~~~
../../binutils-gdb/libctf/ctf-create.c:803:59: note: each undeclared identifier is reported only once for each function it appears in
dtd->dtd_data.ctt_size = clp2 (P2ROUNDUP (ep->cte_bits, NBBY) / NBBY);
^~~~
../../binutils-gdb/libctf/ctf-impl.h:254:42: note: in definition of macro 'P2ROUNDUP'
#define P2ROUNDUP(x, align) (-(-(x) & -(align)))
^~~~~
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_slice':
../../binutils-gdb/libctf/ctf-create.c:862:59: error: 'NBBY' undeclared (first use in this function)
dtd->dtd_data.ctt_size = clp2 (P2ROUNDUP (ep->cte_bits, NBBY) / NBBY);
^~~~
../../binutils-gdb/libctf/ctf-impl.h:254:42: note: in definition of macro 'P2ROUNDUP'
#define P2ROUNDUP(x, align) (-(-(x) & -(align)))
^~~~~
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_member_offset':
../../binutils-gdb/libctf/ctf-create.c:1341:21: error: 'NBBY' undeclared (first use in this function)
off += lsize * NBBY;
^~~~
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_type':
../../binutils-gdb/libctf/ctf-create.c:1822:16: warning: unknown conversion type character 'z' in format [-Wformat=]
ctf_dprintf ("Conflict for type %s against ID %lx: "
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../binutils-gdb/libctf/ctf-create.c:1823:35: note: format string is defined here
"union size differs, old %zi, new %zi\n",
^
../../binutils-gdb/libctf/ctf-create.c:1822:16: warning: unknown conversion type character 'z' in format [-Wformat=]
ctf_dprintf ("Conflict for type %s against ID %lx: "
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../binutils-gdb/libctf/ctf-create.c:1823:44: note: format string is defined here
"union size differs, old %zi, new %zi\n",
^
../../binutils-gdb/libctf/ctf-create.c:1822:16: warning: too many arguments for format [-Wformat-extra-args]
ctf_dprintf ("Conflict for type %s against ID %lx: "
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This patch fixes the actual errors in here. I did not try to fix the
printf warnings, though I think someone ought to.
Ok?
libctf/ChangeLog
2019-06-04 Tom Tromey <tromey@adacore.com>
* ctf-create.c (ctf_add_encoded, ctf_add_slice)
(ctf_add_member_offset): Use CHAR_BIT, not NBBY.
2019-06-05 02:16:57 +08:00
|
|
|
off += lsize * CHAR_BIT;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
|
|
|
/* Round up the offset of the end of the last member to
|
|
|
|
the next byte boundary, convert 'off' to bytes, and
|
|
|
|
then round it up again to the next multiple of the
|
|
|
|
alignment required by the new member. Finally,
|
|
|
|
convert back to bits and store the result in
|
|
|
|
dmd_offset. Technically we could do more efficient
|
|
|
|
packing if the new member is a bit-field, but we're
|
|
|
|
the "compiler" and ANSI says we can do as we choose. */
|
|
|
|
|
Use CHAR_BIT instead of NBBY in libctf
On x86-64 Fedora 29, I tried to build a mingw-hosted gdb that targets
ppc-linux. You can do this with:
../binutils-gdb/configure --host=i686-w64-mingw32 --target=ppc-linux \
--disable-{binutils,gas,gold,gprof,ld}
The build failed with these errors in libctf:
In file included from ../../binutils-gdb/libctf/ctf-create.c:20:
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_encoded':
../../binutils-gdb/libctf/ctf-create.c:803:59: error: 'NBBY' undeclared (first use in this function)
dtd->dtd_data.ctt_size = clp2 (P2ROUNDUP (ep->cte_bits, NBBY) / NBBY);
^~~~
../../binutils-gdb/libctf/ctf-impl.h:254:42: note: in definition of macro 'P2ROUNDUP'
#define P2ROUNDUP(x, align) (-(-(x) & -(align)))
^~~~~
../../binutils-gdb/libctf/ctf-create.c:803:59: note: each undeclared identifier is reported only once for each function it appears in
dtd->dtd_data.ctt_size = clp2 (P2ROUNDUP (ep->cte_bits, NBBY) / NBBY);
^~~~
../../binutils-gdb/libctf/ctf-impl.h:254:42: note: in definition of macro 'P2ROUNDUP'
#define P2ROUNDUP(x, align) (-(-(x) & -(align)))
^~~~~
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_slice':
../../binutils-gdb/libctf/ctf-create.c:862:59: error: 'NBBY' undeclared (first use in this function)
dtd->dtd_data.ctt_size = clp2 (P2ROUNDUP (ep->cte_bits, NBBY) / NBBY);
^~~~
../../binutils-gdb/libctf/ctf-impl.h:254:42: note: in definition of macro 'P2ROUNDUP'
#define P2ROUNDUP(x, align) (-(-(x) & -(align)))
^~~~~
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_member_offset':
../../binutils-gdb/libctf/ctf-create.c:1341:21: error: 'NBBY' undeclared (first use in this function)
off += lsize * NBBY;
^~~~
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_type':
../../binutils-gdb/libctf/ctf-create.c:1822:16: warning: unknown conversion type character 'z' in format [-Wformat=]
ctf_dprintf ("Conflict for type %s against ID %lx: "
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../binutils-gdb/libctf/ctf-create.c:1823:35: note: format string is defined here
"union size differs, old %zi, new %zi\n",
^
../../binutils-gdb/libctf/ctf-create.c:1822:16: warning: unknown conversion type character 'z' in format [-Wformat=]
ctf_dprintf ("Conflict for type %s against ID %lx: "
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../binutils-gdb/libctf/ctf-create.c:1823:44: note: format string is defined here
"union size differs, old %zi, new %zi\n",
^
../../binutils-gdb/libctf/ctf-create.c:1822:16: warning: too many arguments for format [-Wformat-extra-args]
ctf_dprintf ("Conflict for type %s against ID %lx: "
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This patch fixes the actual errors in here. I did not try to fix the
printf warnings, though I think someone ought to.
Ok?
libctf/ChangeLog
2019-06-04 Tom Tromey <tromey@adacore.com>
* ctf-create.c (ctf_add_encoded, ctf_add_slice)
(ctf_add_member_offset): Use CHAR_BIT, not NBBY.
2019-06-05 02:16:57 +08:00
|
|
|
off = roundup (off, CHAR_BIT) / CHAR_BIT;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
off = roundup (off, MAX (malign, 1));
|
Use CHAR_BIT instead of NBBY in libctf
On x86-64 Fedora 29, I tried to build a mingw-hosted gdb that targets
ppc-linux. You can do this with:
../binutils-gdb/configure --host=i686-w64-mingw32 --target=ppc-linux \
--disable-{binutils,gas,gold,gprof,ld}
The build failed with these errors in libctf:
In file included from ../../binutils-gdb/libctf/ctf-create.c:20:
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_encoded':
../../binutils-gdb/libctf/ctf-create.c:803:59: error: 'NBBY' undeclared (first use in this function)
dtd->dtd_data.ctt_size = clp2 (P2ROUNDUP (ep->cte_bits, NBBY) / NBBY);
^~~~
../../binutils-gdb/libctf/ctf-impl.h:254:42: note: in definition of macro 'P2ROUNDUP'
#define P2ROUNDUP(x, align) (-(-(x) & -(align)))
^~~~~
../../binutils-gdb/libctf/ctf-create.c:803:59: note: each undeclared identifier is reported only once for each function it appears in
dtd->dtd_data.ctt_size = clp2 (P2ROUNDUP (ep->cte_bits, NBBY) / NBBY);
^~~~
../../binutils-gdb/libctf/ctf-impl.h:254:42: note: in definition of macro 'P2ROUNDUP'
#define P2ROUNDUP(x, align) (-(-(x) & -(align)))
^~~~~
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_slice':
../../binutils-gdb/libctf/ctf-create.c:862:59: error: 'NBBY' undeclared (first use in this function)
dtd->dtd_data.ctt_size = clp2 (P2ROUNDUP (ep->cte_bits, NBBY) / NBBY);
^~~~
../../binutils-gdb/libctf/ctf-impl.h:254:42: note: in definition of macro 'P2ROUNDUP'
#define P2ROUNDUP(x, align) (-(-(x) & -(align)))
^~~~~
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_member_offset':
../../binutils-gdb/libctf/ctf-create.c:1341:21: error: 'NBBY' undeclared (first use in this function)
off += lsize * NBBY;
^~~~
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_type':
../../binutils-gdb/libctf/ctf-create.c:1822:16: warning: unknown conversion type character 'z' in format [-Wformat=]
ctf_dprintf ("Conflict for type %s against ID %lx: "
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../binutils-gdb/libctf/ctf-create.c:1823:35: note: format string is defined here
"union size differs, old %zi, new %zi\n",
^
../../binutils-gdb/libctf/ctf-create.c:1822:16: warning: unknown conversion type character 'z' in format [-Wformat=]
ctf_dprintf ("Conflict for type %s against ID %lx: "
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../binutils-gdb/libctf/ctf-create.c:1823:44: note: format string is defined here
"union size differs, old %zi, new %zi\n",
^
../../binutils-gdb/libctf/ctf-create.c:1822:16: warning: too many arguments for format [-Wformat-extra-args]
ctf_dprintf ("Conflict for type %s against ID %lx: "
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This patch fixes the actual errors in here. I did not try to fix the
printf warnings, though I think someone ought to.
Ok?
libctf/ChangeLog
2019-06-04 Tom Tromey <tromey@adacore.com>
* ctf-create.c (ctf_add_encoded, ctf_add_slice)
(ctf_add_member_offset): Use CHAR_BIT, not NBBY.
2019-06-05 02:16:57 +08:00
|
|
|
dmd->dmd_offset = off * CHAR_BIT;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
ssize = off + msize;
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
/* Specified offset in bits. */
|
|
|
|
|
|
|
|
dmd->dmd_offset = bit_offset;
|
|
|
|
ssize = ctf_get_ctt_size (fp, &dtd->dtd_data, NULL, NULL);
|
Use CHAR_BIT instead of NBBY in libctf
On x86-64 Fedora 29, I tried to build a mingw-hosted gdb that targets
ppc-linux. You can do this with:
../binutils-gdb/configure --host=i686-w64-mingw32 --target=ppc-linux \
--disable-{binutils,gas,gold,gprof,ld}
The build failed with these errors in libctf:
In file included from ../../binutils-gdb/libctf/ctf-create.c:20:
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_encoded':
../../binutils-gdb/libctf/ctf-create.c:803:59: error: 'NBBY' undeclared (first use in this function)
dtd->dtd_data.ctt_size = clp2 (P2ROUNDUP (ep->cte_bits, NBBY) / NBBY);
^~~~
../../binutils-gdb/libctf/ctf-impl.h:254:42: note: in definition of macro 'P2ROUNDUP'
#define P2ROUNDUP(x, align) (-(-(x) & -(align)))
^~~~~
../../binutils-gdb/libctf/ctf-create.c:803:59: note: each undeclared identifier is reported only once for each function it appears in
dtd->dtd_data.ctt_size = clp2 (P2ROUNDUP (ep->cte_bits, NBBY) / NBBY);
^~~~
../../binutils-gdb/libctf/ctf-impl.h:254:42: note: in definition of macro 'P2ROUNDUP'
#define P2ROUNDUP(x, align) (-(-(x) & -(align)))
^~~~~
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_slice':
../../binutils-gdb/libctf/ctf-create.c:862:59: error: 'NBBY' undeclared (first use in this function)
dtd->dtd_data.ctt_size = clp2 (P2ROUNDUP (ep->cte_bits, NBBY) / NBBY);
^~~~
../../binutils-gdb/libctf/ctf-impl.h:254:42: note: in definition of macro 'P2ROUNDUP'
#define P2ROUNDUP(x, align) (-(-(x) & -(align)))
^~~~~
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_member_offset':
../../binutils-gdb/libctf/ctf-create.c:1341:21: error: 'NBBY' undeclared (first use in this function)
off += lsize * NBBY;
^~~~
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_type':
../../binutils-gdb/libctf/ctf-create.c:1822:16: warning: unknown conversion type character 'z' in format [-Wformat=]
ctf_dprintf ("Conflict for type %s against ID %lx: "
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../binutils-gdb/libctf/ctf-create.c:1823:35: note: format string is defined here
"union size differs, old %zi, new %zi\n",
^
../../binutils-gdb/libctf/ctf-create.c:1822:16: warning: unknown conversion type character 'z' in format [-Wformat=]
ctf_dprintf ("Conflict for type %s against ID %lx: "
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../binutils-gdb/libctf/ctf-create.c:1823:44: note: format string is defined here
"union size differs, old %zi, new %zi\n",
^
../../binutils-gdb/libctf/ctf-create.c:1822:16: warning: too many arguments for format [-Wformat-extra-args]
ctf_dprintf ("Conflict for type %s against ID %lx: "
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This patch fixes the actual errors in here. I did not try to fix the
printf warnings, though I think someone ought to.
Ok?
libctf/ChangeLog
2019-06-04 Tom Tromey <tromey@adacore.com>
* ctf-create.c (ctf_add_encoded, ctf_add_slice)
(ctf_add_member_offset): Use CHAR_BIT, not NBBY.
2019-06-05 02:16:57 +08:00
|
|
|
ssize = MAX (ssize, ((signed) bit_offset / CHAR_BIT) + msize);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
}
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
dmd->dmd_offset = 0;
|
|
|
|
ssize = ctf_get_ctt_size (fp, &dtd->dtd_data, NULL, NULL);
|
|
|
|
ssize = MAX (ssize, msize);
|
|
|
|
}
|
|
|
|
|
libctf: fix a number of build problems found on Solaris and NetBSD
- Use of nonportable <endian.h>
- Use of qsort_r
- Use of zlib without appropriate magic to pull in the binutils zlib
- Use of off64_t without checking (fixed by dropping the unused fields
that need off64_t entirely)
- signedness problems due to long being too short a type on 32-bit
platforms: ctf_id_t is now 'unsigned long', and CTF_ERR must be
used only for functions that return ctf_id_t
- One lingering use of bzero() and of <sys/errno.h>
All fixed, using code from gnulib where possible.
Relatedly, set cts_size in a couple of places it was missed
(string table and symbol table loading upon ctf_bfdopen()).
binutils/
* objdump.c (make_ctfsect): Drop cts_type, cts_flags, and
cts_offset.
* readelf.c (shdr_to_ctf_sect): Likewise.
include/
* ctf-api.h (ctf_sect_t): Drop cts_type, cts_flags, and cts_offset.
(ctf_id_t): This is now an unsigned type.
(CTF_ERR): Cast it to ctf_id_t. Note that it should only be used
for ctf_id_t-returning functions.
libctf/
* Makefile.am (ZLIB): New.
(ZLIBINC): Likewise.
(AM_CFLAGS): Use them.
(libctf_a_LIBADD): New, for LIBOBJS.
* configure.ac: Check for zlib, endian.h, and qsort_r.
* ctf-endian.h: New, providing htole64 and le64toh.
* swap.h: Code style fixes.
(bswap_identity_64): New.
* qsort_r.c: New, from gnulib (with one added #include).
* ctf-decls.h: New, providing a conditional qsort_r declaration,
and unconditional definitions of MIN and MAX.
* ctf-impl.h: Use it. Do not use <sys/errno.h>.
(ctf_set_errno): Now returns unsigned long.
* ctf-util.c (ctf_set_errno): Adjust here too.
* ctf-archive.c: Use ctf-endian.h.
(ctf_arc_open_by_offset): Use memset, not bzero. Drop cts_type,
cts_flags and cts_offset.
(ctf_arc_write): Drop debugging dependent on the size of off_t.
* ctf-create.c: Provide a definition of roundup if not defined.
(ctf_create): Drop cts_type, cts_flags and cts_offset.
(ctf_add_reftype): Do not check if type IDs are below zero.
(ctf_add_slice): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_member_offset): Cast error-returning ssize_t's to size_t
when known error-free. Drop CTF_ERR usage for functions returning
int.
(ctf_add_member_encoded): Drop CTF_ERR usage for functions returning
int.
(ctf_add_variable): Likewise.
(enumcmp): Likewise.
(enumadd): Likewise.
(membcmp): Likewise.
(ctf_add_type): Likewise. Cast error-returning ssize_t's to size_t
when known error-free.
* ctf-dump.c (ctf_is_slice): Drop CTF_ERR usage for functions
returning int: use CTF_ERR for functions returning ctf_type_id.
(ctf_dump_label): Likewise.
(ctf_dump_objts): Likewise.
* ctf-labels.c (ctf_label_topmost): Likewise.
(ctf_label_iter): Likewise.
(ctf_label_info): Likewise.
* ctf-lookup.c (ctf_func_args): Likewise.
* ctf-open.c (upgrade_types): Cast to size_t where appropriate.
(ctf_bufopen): Likewise. Use zlib types as needed.
* ctf-types.c (ctf_member_iter): Drop CTF_ERR usage for functions
returning int.
(ctf_enum_iter): Likewise.
(ctf_type_size): Likewise.
(ctf_type_align): Likewise. Cast to size_t where appropriate.
(ctf_type_kind_unsliced): Likewise.
(ctf_type_kind): Likewise.
(ctf_type_encoding): Likewise.
(ctf_member_info): Likewise.
(ctf_array_info): Likewise.
(ctf_enum_value): Likewise.
(ctf_type_rvisit): Likewise.
* ctf-open-bfd.c (ctf_bfdopen): Drop cts_type, cts_flags and
cts_offset.
(ctf_simple_open): Likewise.
(ctf_bfdopen_ctfsect): Likewise. Set cts_size properly.
* Makefile.in: Regenerate.
* aclocal.m4: Likewise.
* config.h: Likewise.
* configure: Likewise.
2019-05-31 17:10:51 +08:00
|
|
|
if ((size_t) ssize > CTF_MAX_SIZE)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
{
|
|
|
|
dtd->dtd_data.ctt_size = CTF_LSIZE_SENT;
|
|
|
|
dtd->dtd_data.ctt_lsizehi = CTF_SIZE_TO_LSIZE_HI (ssize);
|
|
|
|
dtd->dtd_data.ctt_lsizelo = CTF_SIZE_TO_LSIZE_LO (ssize);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
dtd->dtd_data.ctt_size = (uint32_t) ssize;
|
|
|
|
|
|
|
|
dtd->dtd_data.ctt_info = CTF_TYPE_INFO (kind, root, vlen + 1);
|
|
|
|
ctf_list_append (&dtd->dtd_u.dtu_members, dmd);
|
|
|
|
|
|
|
|
fp->ctf_flags |= LCTF_DIRTY;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
int
|
|
|
|
ctf_add_member_encoded (ctf_file_t *fp, ctf_id_t souid, const char *name,
|
|
|
|
ctf_id_t type, unsigned long bit_offset,
|
|
|
|
const ctf_encoding_t encoding)
|
|
|
|
{
|
|
|
|
ctf_dtdef_t *dtd = ctf_dtd_lookup (fp, type);
|
|
|
|
int kind = LCTF_INFO_KIND (fp, dtd->dtd_data.ctt_info);
|
|
|
|
int otype = type;
|
|
|
|
|
|
|
|
if ((kind != CTF_K_INTEGER) && (kind != CTF_K_FLOAT) && (kind != CTF_K_ENUM))
|
|
|
|
return (ctf_set_errno (fp, ECTF_NOTINTFP));
|
|
|
|
|
|
|
|
if ((type = ctf_add_slice (fp, CTF_ADD_NONROOT, otype, &encoding)) == CTF_ERR)
|
libctf: fix a number of build problems found on Solaris and NetBSD
- Use of nonportable <endian.h>
- Use of qsort_r
- Use of zlib without appropriate magic to pull in the binutils zlib
- Use of off64_t without checking (fixed by dropping the unused fields
that need off64_t entirely)
- signedness problems due to long being too short a type on 32-bit
platforms: ctf_id_t is now 'unsigned long', and CTF_ERR must be
used only for functions that return ctf_id_t
- One lingering use of bzero() and of <sys/errno.h>
All fixed, using code from gnulib where possible.
Relatedly, set cts_size in a couple of places it was missed
(string table and symbol table loading upon ctf_bfdopen()).
binutils/
* objdump.c (make_ctfsect): Drop cts_type, cts_flags, and
cts_offset.
* readelf.c (shdr_to_ctf_sect): Likewise.
include/
* ctf-api.h (ctf_sect_t): Drop cts_type, cts_flags, and cts_offset.
(ctf_id_t): This is now an unsigned type.
(CTF_ERR): Cast it to ctf_id_t. Note that it should only be used
for ctf_id_t-returning functions.
libctf/
* Makefile.am (ZLIB): New.
(ZLIBINC): Likewise.
(AM_CFLAGS): Use them.
(libctf_a_LIBADD): New, for LIBOBJS.
* configure.ac: Check for zlib, endian.h, and qsort_r.
* ctf-endian.h: New, providing htole64 and le64toh.
* swap.h: Code style fixes.
(bswap_identity_64): New.
* qsort_r.c: New, from gnulib (with one added #include).
* ctf-decls.h: New, providing a conditional qsort_r declaration,
and unconditional definitions of MIN and MAX.
* ctf-impl.h: Use it. Do not use <sys/errno.h>.
(ctf_set_errno): Now returns unsigned long.
* ctf-util.c (ctf_set_errno): Adjust here too.
* ctf-archive.c: Use ctf-endian.h.
(ctf_arc_open_by_offset): Use memset, not bzero. Drop cts_type,
cts_flags and cts_offset.
(ctf_arc_write): Drop debugging dependent on the size of off_t.
* ctf-create.c: Provide a definition of roundup if not defined.
(ctf_create): Drop cts_type, cts_flags and cts_offset.
(ctf_add_reftype): Do not check if type IDs are below zero.
(ctf_add_slice): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_member_offset): Cast error-returning ssize_t's to size_t
when known error-free. Drop CTF_ERR usage for functions returning
int.
(ctf_add_member_encoded): Drop CTF_ERR usage for functions returning
int.
(ctf_add_variable): Likewise.
(enumcmp): Likewise.
(enumadd): Likewise.
(membcmp): Likewise.
(ctf_add_type): Likewise. Cast error-returning ssize_t's to size_t
when known error-free.
* ctf-dump.c (ctf_is_slice): Drop CTF_ERR usage for functions
returning int: use CTF_ERR for functions returning ctf_type_id.
(ctf_dump_label): Likewise.
(ctf_dump_objts): Likewise.
* ctf-labels.c (ctf_label_topmost): Likewise.
(ctf_label_iter): Likewise.
(ctf_label_info): Likewise.
* ctf-lookup.c (ctf_func_args): Likewise.
* ctf-open.c (upgrade_types): Cast to size_t where appropriate.
(ctf_bufopen): Likewise. Use zlib types as needed.
* ctf-types.c (ctf_member_iter): Drop CTF_ERR usage for functions
returning int.
(ctf_enum_iter): Likewise.
(ctf_type_size): Likewise.
(ctf_type_align): Likewise. Cast to size_t where appropriate.
(ctf_type_kind_unsliced): Likewise.
(ctf_type_kind): Likewise.
(ctf_type_encoding): Likewise.
(ctf_member_info): Likewise.
(ctf_array_info): Likewise.
(ctf_enum_value): Likewise.
(ctf_type_rvisit): Likewise.
* ctf-open-bfd.c (ctf_bfdopen): Drop cts_type, cts_flags and
cts_offset.
(ctf_simple_open): Likewise.
(ctf_bfdopen_ctfsect): Likewise. Set cts_size properly.
* Makefile.in: Regenerate.
* aclocal.m4: Likewise.
* config.h: Likewise.
* configure: Likewise.
2019-05-31 17:10:51 +08:00
|
|
|
return -1; /* errno is set for us. */
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
|
|
|
return ctf_add_member_offset (fp, souid, name, type, bit_offset);
|
|
|
|
}
|
|
|
|
|
|
|
|
int
|
|
|
|
ctf_add_member (ctf_file_t *fp, ctf_id_t souid, const char *name,
|
|
|
|
ctf_id_t type)
|
|
|
|
{
|
|
|
|
return ctf_add_member_offset (fp, souid, name, type, (unsigned long) - 1);
|
|
|
|
}
|
|
|
|
|
|
|
|
int
|
|
|
|
ctf_add_variable (ctf_file_t *fp, const char *name, ctf_id_t ref)
|
|
|
|
{
|
|
|
|
ctf_dvdef_t *dvd;
|
|
|
|
ctf_file_t *tmp = fp;
|
|
|
|
|
|
|
|
if (!(fp->ctf_flags & LCTF_RDWR))
|
|
|
|
return (ctf_set_errno (fp, ECTF_RDONLY));
|
|
|
|
|
|
|
|
if (ctf_dvd_lookup (fp, name) != NULL)
|
|
|
|
return (ctf_set_errno (fp, ECTF_DUPLICATE));
|
|
|
|
|
|
|
|
if (ctf_lookup_by_id (&tmp, ref) == NULL)
|
libctf: fix a number of build problems found on Solaris and NetBSD
- Use of nonportable <endian.h>
- Use of qsort_r
- Use of zlib without appropriate magic to pull in the binutils zlib
- Use of off64_t without checking (fixed by dropping the unused fields
that need off64_t entirely)
- signedness problems due to long being too short a type on 32-bit
platforms: ctf_id_t is now 'unsigned long', and CTF_ERR must be
used only for functions that return ctf_id_t
- One lingering use of bzero() and of <sys/errno.h>
All fixed, using code from gnulib where possible.
Relatedly, set cts_size in a couple of places it was missed
(string table and symbol table loading upon ctf_bfdopen()).
binutils/
* objdump.c (make_ctfsect): Drop cts_type, cts_flags, and
cts_offset.
* readelf.c (shdr_to_ctf_sect): Likewise.
include/
* ctf-api.h (ctf_sect_t): Drop cts_type, cts_flags, and cts_offset.
(ctf_id_t): This is now an unsigned type.
(CTF_ERR): Cast it to ctf_id_t. Note that it should only be used
for ctf_id_t-returning functions.
libctf/
* Makefile.am (ZLIB): New.
(ZLIBINC): Likewise.
(AM_CFLAGS): Use them.
(libctf_a_LIBADD): New, for LIBOBJS.
* configure.ac: Check for zlib, endian.h, and qsort_r.
* ctf-endian.h: New, providing htole64 and le64toh.
* swap.h: Code style fixes.
(bswap_identity_64): New.
* qsort_r.c: New, from gnulib (with one added #include).
* ctf-decls.h: New, providing a conditional qsort_r declaration,
and unconditional definitions of MIN and MAX.
* ctf-impl.h: Use it. Do not use <sys/errno.h>.
(ctf_set_errno): Now returns unsigned long.
* ctf-util.c (ctf_set_errno): Adjust here too.
* ctf-archive.c: Use ctf-endian.h.
(ctf_arc_open_by_offset): Use memset, not bzero. Drop cts_type,
cts_flags and cts_offset.
(ctf_arc_write): Drop debugging dependent on the size of off_t.
* ctf-create.c: Provide a definition of roundup if not defined.
(ctf_create): Drop cts_type, cts_flags and cts_offset.
(ctf_add_reftype): Do not check if type IDs are below zero.
(ctf_add_slice): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_member_offset): Cast error-returning ssize_t's to size_t
when known error-free. Drop CTF_ERR usage for functions returning
int.
(ctf_add_member_encoded): Drop CTF_ERR usage for functions returning
int.
(ctf_add_variable): Likewise.
(enumcmp): Likewise.
(enumadd): Likewise.
(membcmp): Likewise.
(ctf_add_type): Likewise. Cast error-returning ssize_t's to size_t
when known error-free.
* ctf-dump.c (ctf_is_slice): Drop CTF_ERR usage for functions
returning int: use CTF_ERR for functions returning ctf_type_id.
(ctf_dump_label): Likewise.
(ctf_dump_objts): Likewise.
* ctf-labels.c (ctf_label_topmost): Likewise.
(ctf_label_iter): Likewise.
(ctf_label_info): Likewise.
* ctf-lookup.c (ctf_func_args): Likewise.
* ctf-open.c (upgrade_types): Cast to size_t where appropriate.
(ctf_bufopen): Likewise. Use zlib types as needed.
* ctf-types.c (ctf_member_iter): Drop CTF_ERR usage for functions
returning int.
(ctf_enum_iter): Likewise.
(ctf_type_size): Likewise.
(ctf_type_align): Likewise. Cast to size_t where appropriate.
(ctf_type_kind_unsliced): Likewise.
(ctf_type_kind): Likewise.
(ctf_type_encoding): Likewise.
(ctf_member_info): Likewise.
(ctf_array_info): Likewise.
(ctf_enum_value): Likewise.
(ctf_type_rvisit): Likewise.
* ctf-open-bfd.c (ctf_bfdopen): Drop cts_type, cts_flags and
cts_offset.
(ctf_simple_open): Likewise.
(ctf_bfdopen_ctfsect): Likewise. Set cts_size properly.
* Makefile.in: Regenerate.
* aclocal.m4: Likewise.
* config.h: Likewise.
* configure: Likewise.
2019-05-31 17:10:51 +08:00
|
|
|
return -1; /* errno is set for us. */
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
libctf: handle nonrepresentable types at link time
GCC can emit references to type 0 to indicate that this type is one that
is not representable in the version of CTF it emits (for instance,
version 3 cannot encode vector types). Type 0 is already used in the
function section to indicate padding inserted to skip functions we do
not want to encode the type of, so using zero in this way is a good
extension of the format: but libctf reports such types as ECTF_BADID,
which is indistinguishable from file corruption via links to truly
nonexistent types with IDs like 0xDEADBEEF etc, which we really do want
to stop for.
In particular, this stops all traversals of types dead at this point,
preventing us from even dumping CTF files containing unrepresentable
types to see what's going on!
So add a new error, ECTF_NONREPRESENTABLE, which is returned by
recursive type resolution when a reference to a zero type is found. (No
zero type is ever emitted into the CTF file by GCC, only references to
one). We can't do much with types that are ultimately nonrepresentable,
but we can do enough to keep functioning.
Adjust ctf_add_type to ensure that top-level types of type zero and
structure and union members of ultimate type zero are simply skipped
without reporting an error, so we can copy structures and unions that
contain nonrepresentable members (skipping them and leaving a hole where
they would be, so no consumers downstream of the linker need to worry
about this): adjust the dumper so that we dump members of
nonrepresentable types in a simple form that indicates
nonrepresentability rather than terminating the dump, and do not falsely
assume all errors to be -ENOMEM: adjust the linker so that types that
fail to get added are simply skipped, so that both nonrepresentable
types and outright errors do not terminate the type addition, which
could skip many valid types and cause further errors when variables of
those types are added.
In future, when we gain the ability to call back to the linker to report
link-time type resolution errors, we should report failures to add all
but nonrepresentable types. But we can't do that yet.
v5: Fix tabdamage.
include/
* ctf-api.h (ECTF_NONREPRESENTABLE): New.
libctf/
* ctf-types.c (ctf_type_resolve): Return ECTF_NONREPRESENTABLE on
type zero.
* ctf-create.c (ctf_add_type): Detect and skip nonrepresentable
members and types.
(ctf_add_variable): Likewise for variables pointing to them.
* ctf-link.c (ctf_link_one_type): Do not warn for nonrepresentable
type link failure, but do warn for others.
* ctf-dump.c (ctf_dump_format_type): Likewise. Do not assume all
errors to be ENOMEM.
(ctf_dump_member): Likewise.
(ctf_dump_type): Likewise.
(ctf_dump_header_strfield): Do not assume all errors to be ENOMEM.
(ctf_dump_header_sectfield): Do not assume all errors to be ENOMEM.
(ctf_dump_header): Likewise.
(ctf_dump_label): likewise.
(ctf_dump_objts): likewise.
(ctf_dump_funcs): likewise.
(ctf_dump_var): likewise.
(ctf_dump_str): Likewise.
2019-08-05 18:40:33 +08:00
|
|
|
/* Make sure this type is representable. */
|
|
|
|
if ((ctf_type_resolve (fp, ref) == CTF_ERR)
|
|
|
|
&& (ctf_errno (fp) == ECTF_NONREPRESENTABLE))
|
|
|
|
return -1;
|
|
|
|
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
if ((dvd = malloc (sizeof (ctf_dvdef_t))) == NULL)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return (ctf_set_errno (fp, EAGAIN));
|
|
|
|
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
if (name != NULL && (dvd->dvd_name = strdup (name)) == NULL)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
{
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
free (dvd);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return (ctf_set_errno (fp, EAGAIN));
|
|
|
|
}
|
|
|
|
dvd->dvd_type = ref;
|
|
|
|
dvd->dvd_snapshots = fp->ctf_snapshots;
|
|
|
|
|
2019-06-19 19:14:16 +08:00
|
|
|
if (ctf_dvd_insert (fp, dvd) < 0)
|
|
|
|
{
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
free (dvd->dvd_name);
|
|
|
|
free (dvd);
|
2019-06-19 19:14:16 +08:00
|
|
|
return -1; /* errno is set for us. */
|
|
|
|
}
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
|
|
|
fp->ctf_flags |= LCTF_DIRTY;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2019-04-24 18:22:03 +08:00
|
|
|
static int
|
|
|
|
enumcmp (const char *name, int value, void *arg)
|
|
|
|
{
|
|
|
|
ctf_bundle_t *ctb = arg;
|
|
|
|
int bvalue;
|
|
|
|
|
libctf: fix a number of build problems found on Solaris and NetBSD
- Use of nonportable <endian.h>
- Use of qsort_r
- Use of zlib without appropriate magic to pull in the binutils zlib
- Use of off64_t without checking (fixed by dropping the unused fields
that need off64_t entirely)
- signedness problems due to long being too short a type on 32-bit
platforms: ctf_id_t is now 'unsigned long', and CTF_ERR must be
used only for functions that return ctf_id_t
- One lingering use of bzero() and of <sys/errno.h>
All fixed, using code from gnulib where possible.
Relatedly, set cts_size in a couple of places it was missed
(string table and symbol table loading upon ctf_bfdopen()).
binutils/
* objdump.c (make_ctfsect): Drop cts_type, cts_flags, and
cts_offset.
* readelf.c (shdr_to_ctf_sect): Likewise.
include/
* ctf-api.h (ctf_sect_t): Drop cts_type, cts_flags, and cts_offset.
(ctf_id_t): This is now an unsigned type.
(CTF_ERR): Cast it to ctf_id_t. Note that it should only be used
for ctf_id_t-returning functions.
libctf/
* Makefile.am (ZLIB): New.
(ZLIBINC): Likewise.
(AM_CFLAGS): Use them.
(libctf_a_LIBADD): New, for LIBOBJS.
* configure.ac: Check for zlib, endian.h, and qsort_r.
* ctf-endian.h: New, providing htole64 and le64toh.
* swap.h: Code style fixes.
(bswap_identity_64): New.
* qsort_r.c: New, from gnulib (with one added #include).
* ctf-decls.h: New, providing a conditional qsort_r declaration,
and unconditional definitions of MIN and MAX.
* ctf-impl.h: Use it. Do not use <sys/errno.h>.
(ctf_set_errno): Now returns unsigned long.
* ctf-util.c (ctf_set_errno): Adjust here too.
* ctf-archive.c: Use ctf-endian.h.
(ctf_arc_open_by_offset): Use memset, not bzero. Drop cts_type,
cts_flags and cts_offset.
(ctf_arc_write): Drop debugging dependent on the size of off_t.
* ctf-create.c: Provide a definition of roundup if not defined.
(ctf_create): Drop cts_type, cts_flags and cts_offset.
(ctf_add_reftype): Do not check if type IDs are below zero.
(ctf_add_slice): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_member_offset): Cast error-returning ssize_t's to size_t
when known error-free. Drop CTF_ERR usage for functions returning
int.
(ctf_add_member_encoded): Drop CTF_ERR usage for functions returning
int.
(ctf_add_variable): Likewise.
(enumcmp): Likewise.
(enumadd): Likewise.
(membcmp): Likewise.
(ctf_add_type): Likewise. Cast error-returning ssize_t's to size_t
when known error-free.
* ctf-dump.c (ctf_is_slice): Drop CTF_ERR usage for functions
returning int: use CTF_ERR for functions returning ctf_type_id.
(ctf_dump_label): Likewise.
(ctf_dump_objts): Likewise.
* ctf-labels.c (ctf_label_topmost): Likewise.
(ctf_label_iter): Likewise.
(ctf_label_info): Likewise.
* ctf-lookup.c (ctf_func_args): Likewise.
* ctf-open.c (upgrade_types): Cast to size_t where appropriate.
(ctf_bufopen): Likewise. Use zlib types as needed.
* ctf-types.c (ctf_member_iter): Drop CTF_ERR usage for functions
returning int.
(ctf_enum_iter): Likewise.
(ctf_type_size): Likewise.
(ctf_type_align): Likewise. Cast to size_t where appropriate.
(ctf_type_kind_unsliced): Likewise.
(ctf_type_kind): Likewise.
(ctf_type_encoding): Likewise.
(ctf_member_info): Likewise.
(ctf_array_info): Likewise.
(ctf_enum_value): Likewise.
(ctf_type_rvisit): Likewise.
* ctf-open-bfd.c (ctf_bfdopen): Drop cts_type, cts_flags and
cts_offset.
(ctf_simple_open): Likewise.
(ctf_bfdopen_ctfsect): Likewise. Set cts_size properly.
* Makefile.in: Regenerate.
* aclocal.m4: Likewise.
* config.h: Likewise.
* configure: Likewise.
2019-05-31 17:10:51 +08:00
|
|
|
if (ctf_enum_value (ctb->ctb_file, ctb->ctb_type, name, &bvalue) < 0)
|
2019-04-24 18:22:03 +08:00
|
|
|
{
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
ctf_dprintf ("Conflict due to member %s iteration error: %s.\n", name,
|
|
|
|
ctf_errmsg (ctf_errno (ctb->ctb_file)));
|
2019-04-24 18:22:03 +08:00
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
if (value != bvalue)
|
|
|
|
{
|
|
|
|
ctf_dprintf ("Conflict due to value change: %i versus %i\n",
|
|
|
|
value, bvalue);
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int
|
|
|
|
enumadd (const char *name, int value, void *arg)
|
|
|
|
{
|
|
|
|
ctf_bundle_t *ctb = arg;
|
|
|
|
|
|
|
|
return (ctf_add_enumerator (ctb->ctb_file, ctb->ctb_type,
|
libctf: fix a number of build problems found on Solaris and NetBSD
- Use of nonportable <endian.h>
- Use of qsort_r
- Use of zlib without appropriate magic to pull in the binutils zlib
- Use of off64_t without checking (fixed by dropping the unused fields
that need off64_t entirely)
- signedness problems due to long being too short a type on 32-bit
platforms: ctf_id_t is now 'unsigned long', and CTF_ERR must be
used only for functions that return ctf_id_t
- One lingering use of bzero() and of <sys/errno.h>
All fixed, using code from gnulib where possible.
Relatedly, set cts_size in a couple of places it was missed
(string table and symbol table loading upon ctf_bfdopen()).
binutils/
* objdump.c (make_ctfsect): Drop cts_type, cts_flags, and
cts_offset.
* readelf.c (shdr_to_ctf_sect): Likewise.
include/
* ctf-api.h (ctf_sect_t): Drop cts_type, cts_flags, and cts_offset.
(ctf_id_t): This is now an unsigned type.
(CTF_ERR): Cast it to ctf_id_t. Note that it should only be used
for ctf_id_t-returning functions.
libctf/
* Makefile.am (ZLIB): New.
(ZLIBINC): Likewise.
(AM_CFLAGS): Use them.
(libctf_a_LIBADD): New, for LIBOBJS.
* configure.ac: Check for zlib, endian.h, and qsort_r.
* ctf-endian.h: New, providing htole64 and le64toh.
* swap.h: Code style fixes.
(bswap_identity_64): New.
* qsort_r.c: New, from gnulib (with one added #include).
* ctf-decls.h: New, providing a conditional qsort_r declaration,
and unconditional definitions of MIN and MAX.
* ctf-impl.h: Use it. Do not use <sys/errno.h>.
(ctf_set_errno): Now returns unsigned long.
* ctf-util.c (ctf_set_errno): Adjust here too.
* ctf-archive.c: Use ctf-endian.h.
(ctf_arc_open_by_offset): Use memset, not bzero. Drop cts_type,
cts_flags and cts_offset.
(ctf_arc_write): Drop debugging dependent on the size of off_t.
* ctf-create.c: Provide a definition of roundup if not defined.
(ctf_create): Drop cts_type, cts_flags and cts_offset.
(ctf_add_reftype): Do not check if type IDs are below zero.
(ctf_add_slice): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_member_offset): Cast error-returning ssize_t's to size_t
when known error-free. Drop CTF_ERR usage for functions returning
int.
(ctf_add_member_encoded): Drop CTF_ERR usage for functions returning
int.
(ctf_add_variable): Likewise.
(enumcmp): Likewise.
(enumadd): Likewise.
(membcmp): Likewise.
(ctf_add_type): Likewise. Cast error-returning ssize_t's to size_t
when known error-free.
* ctf-dump.c (ctf_is_slice): Drop CTF_ERR usage for functions
returning int: use CTF_ERR for functions returning ctf_type_id.
(ctf_dump_label): Likewise.
(ctf_dump_objts): Likewise.
* ctf-labels.c (ctf_label_topmost): Likewise.
(ctf_label_iter): Likewise.
(ctf_label_info): Likewise.
* ctf-lookup.c (ctf_func_args): Likewise.
* ctf-open.c (upgrade_types): Cast to size_t where appropriate.
(ctf_bufopen): Likewise. Use zlib types as needed.
* ctf-types.c (ctf_member_iter): Drop CTF_ERR usage for functions
returning int.
(ctf_enum_iter): Likewise.
(ctf_type_size): Likewise.
(ctf_type_align): Likewise. Cast to size_t where appropriate.
(ctf_type_kind_unsliced): Likewise.
(ctf_type_kind): Likewise.
(ctf_type_encoding): Likewise.
(ctf_member_info): Likewise.
(ctf_array_info): Likewise.
(ctf_enum_value): Likewise.
(ctf_type_rvisit): Likewise.
* ctf-open-bfd.c (ctf_bfdopen): Drop cts_type, cts_flags and
cts_offset.
(ctf_simple_open): Likewise.
(ctf_bfdopen_ctfsect): Likewise. Set cts_size properly.
* Makefile.in: Regenerate.
* aclocal.m4: Likewise.
* config.h: Likewise.
* configure: Likewise.
2019-05-31 17:10:51 +08:00
|
|
|
name, value) < 0);
|
2019-04-24 18:22:03 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static int
|
|
|
|
membcmp (const char *name, ctf_id_t type _libctf_unused_, unsigned long offset,
|
|
|
|
void *arg)
|
|
|
|
{
|
|
|
|
ctf_bundle_t *ctb = arg;
|
|
|
|
ctf_membinfo_t ctm;
|
|
|
|
|
2020-06-03 03:38:17 +08:00
|
|
|
/* Don't check nameless members (e.g. anonymous structs/unions) against each
|
|
|
|
other. */
|
|
|
|
if (name[0] == 0)
|
|
|
|
return 0;
|
|
|
|
|
libctf: fix a number of build problems found on Solaris and NetBSD
- Use of nonportable <endian.h>
- Use of qsort_r
- Use of zlib without appropriate magic to pull in the binutils zlib
- Use of off64_t without checking (fixed by dropping the unused fields
that need off64_t entirely)
- signedness problems due to long being too short a type on 32-bit
platforms: ctf_id_t is now 'unsigned long', and CTF_ERR must be
used only for functions that return ctf_id_t
- One lingering use of bzero() and of <sys/errno.h>
All fixed, using code from gnulib where possible.
Relatedly, set cts_size in a couple of places it was missed
(string table and symbol table loading upon ctf_bfdopen()).
binutils/
* objdump.c (make_ctfsect): Drop cts_type, cts_flags, and
cts_offset.
* readelf.c (shdr_to_ctf_sect): Likewise.
include/
* ctf-api.h (ctf_sect_t): Drop cts_type, cts_flags, and cts_offset.
(ctf_id_t): This is now an unsigned type.
(CTF_ERR): Cast it to ctf_id_t. Note that it should only be used
for ctf_id_t-returning functions.
libctf/
* Makefile.am (ZLIB): New.
(ZLIBINC): Likewise.
(AM_CFLAGS): Use them.
(libctf_a_LIBADD): New, for LIBOBJS.
* configure.ac: Check for zlib, endian.h, and qsort_r.
* ctf-endian.h: New, providing htole64 and le64toh.
* swap.h: Code style fixes.
(bswap_identity_64): New.
* qsort_r.c: New, from gnulib (with one added #include).
* ctf-decls.h: New, providing a conditional qsort_r declaration,
and unconditional definitions of MIN and MAX.
* ctf-impl.h: Use it. Do not use <sys/errno.h>.
(ctf_set_errno): Now returns unsigned long.
* ctf-util.c (ctf_set_errno): Adjust here too.
* ctf-archive.c: Use ctf-endian.h.
(ctf_arc_open_by_offset): Use memset, not bzero. Drop cts_type,
cts_flags and cts_offset.
(ctf_arc_write): Drop debugging dependent on the size of off_t.
* ctf-create.c: Provide a definition of roundup if not defined.
(ctf_create): Drop cts_type, cts_flags and cts_offset.
(ctf_add_reftype): Do not check if type IDs are below zero.
(ctf_add_slice): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_member_offset): Cast error-returning ssize_t's to size_t
when known error-free. Drop CTF_ERR usage for functions returning
int.
(ctf_add_member_encoded): Drop CTF_ERR usage for functions returning
int.
(ctf_add_variable): Likewise.
(enumcmp): Likewise.
(enumadd): Likewise.
(membcmp): Likewise.
(ctf_add_type): Likewise. Cast error-returning ssize_t's to size_t
when known error-free.
* ctf-dump.c (ctf_is_slice): Drop CTF_ERR usage for functions
returning int: use CTF_ERR for functions returning ctf_type_id.
(ctf_dump_label): Likewise.
(ctf_dump_objts): Likewise.
* ctf-labels.c (ctf_label_topmost): Likewise.
(ctf_label_iter): Likewise.
(ctf_label_info): Likewise.
* ctf-lookup.c (ctf_func_args): Likewise.
* ctf-open.c (upgrade_types): Cast to size_t where appropriate.
(ctf_bufopen): Likewise. Use zlib types as needed.
* ctf-types.c (ctf_member_iter): Drop CTF_ERR usage for functions
returning int.
(ctf_enum_iter): Likewise.
(ctf_type_size): Likewise.
(ctf_type_align): Likewise. Cast to size_t where appropriate.
(ctf_type_kind_unsliced): Likewise.
(ctf_type_kind): Likewise.
(ctf_type_encoding): Likewise.
(ctf_member_info): Likewise.
(ctf_array_info): Likewise.
(ctf_enum_value): Likewise.
(ctf_type_rvisit): Likewise.
* ctf-open-bfd.c (ctf_bfdopen): Drop cts_type, cts_flags and
cts_offset.
(ctf_simple_open): Likewise.
(ctf_bfdopen_ctfsect): Likewise. Set cts_size properly.
* Makefile.in: Regenerate.
* aclocal.m4: Likewise.
* config.h: Likewise.
* configure: Likewise.
2019-05-31 17:10:51 +08:00
|
|
|
if (ctf_member_info (ctb->ctb_file, ctb->ctb_type, name, &ctm) < 0)
|
2019-04-24 18:22:03 +08:00
|
|
|
{
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
ctf_dprintf ("Conflict due to member %s iteration error: %s.\n", name,
|
|
|
|
ctf_errmsg (ctf_errno (ctb->ctb_file)));
|
2019-04-24 18:22:03 +08:00
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
if (ctm.ctm_offset != offset)
|
|
|
|
{
|
|
|
|
ctf_dprintf ("Conflict due to member %s offset change: "
|
|
|
|
"%lx versus %lx\n", name, ctm.ctm_offset, offset);
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int
|
|
|
|
membadd (const char *name, ctf_id_t type, unsigned long offset, void *arg)
|
|
|
|
{
|
|
|
|
ctf_bundle_t *ctb = arg;
|
|
|
|
ctf_dmdef_t *dmd;
|
|
|
|
char *s = NULL;
|
|
|
|
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
if ((dmd = malloc (sizeof (ctf_dmdef_t))) == NULL)
|
2019-04-24 18:22:03 +08:00
|
|
|
return (ctf_set_errno (ctb->ctb_file, EAGAIN));
|
|
|
|
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
if (name != NULL && (s = strdup (name)) == NULL)
|
2019-04-24 18:22:03 +08:00
|
|
|
{
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
free (dmd);
|
2019-04-24 18:22:03 +08:00
|
|
|
return (ctf_set_errno (ctb->ctb_file, EAGAIN));
|
|
|
|
}
|
|
|
|
|
|
|
|
/* For now, dmd_type is copied as the src_fp's type; it is reset to an
|
|
|
|
equivalent dst_fp type by a final loop in ctf_add_type(), below. */
|
|
|
|
dmd->dmd_name = s;
|
|
|
|
dmd->dmd_type = type;
|
|
|
|
dmd->dmd_offset = offset;
|
|
|
|
dmd->dmd_value = -1;
|
|
|
|
|
|
|
|
ctf_list_append (&ctb->ctb_dtd->dtd_u.dtu_members, dmd);
|
|
|
|
|
|
|
|
ctb->ctb_file->ctf_flags |= LCTF_DIRTY;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* The ctf_add_type routine is used to copy a type from a source CTF container
|
|
|
|
to a dynamic destination container. This routine operates recursively by
|
|
|
|
following the source type's links and embedded member types. If the
|
|
|
|
destination container already contains a named type which has the same
|
|
|
|
attributes, then we succeed and return this type but no changes occur. */
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
static ctf_id_t
|
|
|
|
ctf_add_type_internal (ctf_file_t *dst_fp, ctf_file_t *src_fp, ctf_id_t src_type,
|
|
|
|
ctf_file_t *proc_tracking_fp)
|
2019-04-24 18:22:03 +08:00
|
|
|
{
|
|
|
|
ctf_id_t dst_type = CTF_ERR;
|
|
|
|
uint32_t dst_kind = CTF_K_UNKNOWN;
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
ctf_file_t *tmp_fp = dst_fp;
|
2019-04-24 18:22:03 +08:00
|
|
|
ctf_id_t tmp;
|
|
|
|
|
|
|
|
const char *name;
|
libctf: teach ctf_add_type how forwards work
This machinery has been broken for as long as Solaris has existed.
Forwards are meant to encode "struct foo;", "enum foo;" or "union
foo;". Obviously these all exist in distinct namespaces, so forwards
store the type kind they forward to in their ctt_type member
(which makes conceptual sense if you squint at it). The addition
machinery uses this to promote forwards to the appropriate type as
needed.
Unfortunately ctf_add_type does not: it checks the global namespace
(which is always wrong), and so fails with a spurious conflict if you
have, say, a typedef and then a forward comes along with the same name,
even if it's a forward to something like a struct. (This was observed
with <libio.h>, which has "struct _IO_FILE;" and also
"typedef struct _IO_FILE _IO_FILE"). We should look at the recorded
type kind and look in the appropriate namespace. We should also,
when creating the forward in the new container, use that type kind,
rather than just defaulting to CTF_K_STRUCT and hoping that what
eventually comes along is a struct.
This bug is as old as the first implementation of ctf_add_type in
Solaris. But we also want a new feature for the linker, closely-related
and touching the same code so we add it here: not only do we want a
forward followed by a struct/union/enum to promote the forward, but
we want want a struct/union/enum followed by a forward to act as a NOP
and return the existing type, because when we're adding many files
in succession to a target link, there will often be already-promoted
forwards (in the shape of a struct/union/enum) that want to unify
with duplicate forwards coming from other object files.
v5: fix tabdamage.
libctf/
* ctf-create.c (ctf_add_type): Look up and use the forwarded-to
type kind. Allow forwards to unify with pre-existing structs/
unions/enums.
2019-08-03 07:46:01 +08:00
|
|
|
uint32_t kind, forward_kind, flag, vlen;
|
2019-04-24 18:22:03 +08:00
|
|
|
|
|
|
|
const ctf_type_t *src_tp, *dst_tp;
|
|
|
|
ctf_bundle_t src, dst;
|
|
|
|
ctf_encoding_t src_en, dst_en;
|
|
|
|
ctf_arinfo_t src_ar, dst_ar;
|
|
|
|
|
|
|
|
ctf_funcinfo_t ctc;
|
|
|
|
|
2019-07-14 04:31:26 +08:00
|
|
|
ctf_id_t orig_src_type = src_type;
|
2019-04-24 18:22:03 +08:00
|
|
|
|
|
|
|
if (!(dst_fp->ctf_flags & LCTF_RDWR))
|
|
|
|
return (ctf_set_errno (dst_fp, ECTF_RDONLY));
|
|
|
|
|
|
|
|
if ((src_tp = ctf_lookup_by_id (&src_fp, src_type)) == NULL)
|
|
|
|
return (ctf_set_errno (dst_fp, ctf_errno (src_fp)));
|
|
|
|
|
libctf: handle nonrepresentable types at link time
GCC can emit references to type 0 to indicate that this type is one that
is not representable in the version of CTF it emits (for instance,
version 3 cannot encode vector types). Type 0 is already used in the
function section to indicate padding inserted to skip functions we do
not want to encode the type of, so using zero in this way is a good
extension of the format: but libctf reports such types as ECTF_BADID,
which is indistinguishable from file corruption via links to truly
nonexistent types with IDs like 0xDEADBEEF etc, which we really do want
to stop for.
In particular, this stops all traversals of types dead at this point,
preventing us from even dumping CTF files containing unrepresentable
types to see what's going on!
So add a new error, ECTF_NONREPRESENTABLE, which is returned by
recursive type resolution when a reference to a zero type is found. (No
zero type is ever emitted into the CTF file by GCC, only references to
one). We can't do much with types that are ultimately nonrepresentable,
but we can do enough to keep functioning.
Adjust ctf_add_type to ensure that top-level types of type zero and
structure and union members of ultimate type zero are simply skipped
without reporting an error, so we can copy structures and unions that
contain nonrepresentable members (skipping them and leaving a hole where
they would be, so no consumers downstream of the linker need to worry
about this): adjust the dumper so that we dump members of
nonrepresentable types in a simple form that indicates
nonrepresentability rather than terminating the dump, and do not falsely
assume all errors to be -ENOMEM: adjust the linker so that types that
fail to get added are simply skipped, so that both nonrepresentable
types and outright errors do not terminate the type addition, which
could skip many valid types and cause further errors when variables of
those types are added.
In future, when we gain the ability to call back to the linker to report
link-time type resolution errors, we should report failures to add all
but nonrepresentable types. But we can't do that yet.
v5: Fix tabdamage.
include/
* ctf-api.h (ECTF_NONREPRESENTABLE): New.
libctf/
* ctf-types.c (ctf_type_resolve): Return ECTF_NONREPRESENTABLE on
type zero.
* ctf-create.c (ctf_add_type): Detect and skip nonrepresentable
members and types.
(ctf_add_variable): Likewise for variables pointing to them.
* ctf-link.c (ctf_link_one_type): Do not warn for nonrepresentable
type link failure, but do warn for others.
* ctf-dump.c (ctf_dump_format_type): Likewise. Do not assume all
errors to be ENOMEM.
(ctf_dump_member): Likewise.
(ctf_dump_type): Likewise.
(ctf_dump_header_strfield): Do not assume all errors to be ENOMEM.
(ctf_dump_header_sectfield): Do not assume all errors to be ENOMEM.
(ctf_dump_header): Likewise.
(ctf_dump_label): likewise.
(ctf_dump_objts): likewise.
(ctf_dump_funcs): likewise.
(ctf_dump_var): likewise.
(ctf_dump_str): Likewise.
2019-08-05 18:40:33 +08:00
|
|
|
if ((ctf_type_resolve (src_fp, src_type) == CTF_ERR)
|
|
|
|
&& (ctf_errno (src_fp) == ECTF_NONREPRESENTABLE))
|
|
|
|
return (ctf_set_errno (dst_fp, ECTF_NONREPRESENTABLE));
|
|
|
|
|
2019-04-24 18:22:03 +08:00
|
|
|
name = ctf_strptr (src_fp, src_tp->ctt_name);
|
|
|
|
kind = LCTF_INFO_KIND (src_fp, src_tp->ctt_info);
|
|
|
|
flag = LCTF_INFO_ISROOT (src_fp, src_tp->ctt_info);
|
|
|
|
vlen = LCTF_INFO_VLEN (src_fp, src_tp->ctt_info);
|
|
|
|
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
/* If this is a type we are currently in the middle of adding, hand it
|
|
|
|
straight back. (This lets us handle self-referential structures without
|
|
|
|
considering forwards and empty structures the same as their completed
|
|
|
|
forms.) */
|
|
|
|
|
|
|
|
tmp = ctf_type_mapping (src_fp, src_type, &tmp_fp);
|
|
|
|
|
|
|
|
if (tmp != 0)
|
|
|
|
{
|
|
|
|
if (ctf_dynhash_lookup (proc_tracking_fp->ctf_add_processing,
|
|
|
|
(void *) (uintptr_t) src_type))
|
|
|
|
return tmp;
|
|
|
|
|
|
|
|
/* If this type has already been added from this container, and is the same
|
|
|
|
kind and (if a struct or union) has the same number of members, hand it
|
|
|
|
straight back. */
|
|
|
|
|
2019-11-05 21:09:57 +08:00
|
|
|
if (ctf_type_kind_unsliced (tmp_fp, tmp) == (int) kind)
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
{
|
2019-11-05 21:09:57 +08:00
|
|
|
if (kind == CTF_K_STRUCT || kind == CTF_K_UNION
|
|
|
|
|| kind == CTF_K_ENUM)
|
|
|
|
{
|
|
|
|
if ((dst_tp = ctf_lookup_by_id (&tmp_fp, dst_type)) != NULL)
|
|
|
|
if (vlen == LCTF_INFO_VLEN (tmp_fp, dst_tp->ctt_info))
|
|
|
|
return tmp;
|
|
|
|
}
|
|
|
|
else
|
|
|
|
return tmp;
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
libctf: teach ctf_add_type how forwards work
This machinery has been broken for as long as Solaris has existed.
Forwards are meant to encode "struct foo;", "enum foo;" or "union
foo;". Obviously these all exist in distinct namespaces, so forwards
store the type kind they forward to in their ctt_type member
(which makes conceptual sense if you squint at it). The addition
machinery uses this to promote forwards to the appropriate type as
needed.
Unfortunately ctf_add_type does not: it checks the global namespace
(which is always wrong), and so fails with a spurious conflict if you
have, say, a typedef and then a forward comes along with the same name,
even if it's a forward to something like a struct. (This was observed
with <libio.h>, which has "struct _IO_FILE;" and also
"typedef struct _IO_FILE _IO_FILE"). We should look at the recorded
type kind and look in the appropriate namespace. We should also,
when creating the forward in the new container, use that type kind,
rather than just defaulting to CTF_K_STRUCT and hoping that what
eventually comes along is a struct.
This bug is as old as the first implementation of ctf_add_type in
Solaris. But we also want a new feature for the linker, closely-related
and touching the same code so we add it here: not only do we want a
forward followed by a struct/union/enum to promote the forward, but
we want want a struct/union/enum followed by a forward to act as a NOP
and return the existing type, because when we're adding many files
in succession to a target link, there will often be already-promoted
forwards (in the shape of a struct/union/enum) that want to unify
with duplicate forwards coming from other object files.
v5: fix tabdamage.
libctf/
* ctf-create.c (ctf_add_type): Look up and use the forwarded-to
type kind. Allow forwards to unify with pre-existing structs/
unions/enums.
2019-08-03 07:46:01 +08:00
|
|
|
forward_kind = kind;
|
|
|
|
if (kind == CTF_K_FORWARD)
|
|
|
|
forward_kind = src_tp->ctt_type;
|
|
|
|
|
2019-04-24 18:22:03 +08:00
|
|
|
/* If the source type has a name and is a root type (visible at the
|
|
|
|
top-level scope), lookup the name in the destination container and
|
|
|
|
verify that it is of the same kind before we do anything else. */
|
|
|
|
|
|
|
|
if ((flag & CTF_ADD_ROOT) && name[0] != '\0'
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
&& (tmp = ctf_lookup_by_rawname (dst_fp, forward_kind, name)) != 0)
|
2019-04-24 18:22:03 +08:00
|
|
|
{
|
|
|
|
dst_type = tmp;
|
|
|
|
dst_kind = ctf_type_kind_unsliced (dst_fp, dst_type);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* If an identically named dst_type exists, fail with ECTF_CONFLICT
|
|
|
|
unless dst_type is a forward declaration and src_type is a struct,
|
libctf: teach ctf_add_type how forwards work
This machinery has been broken for as long as Solaris has existed.
Forwards are meant to encode "struct foo;", "enum foo;" or "union
foo;". Obviously these all exist in distinct namespaces, so forwards
store the type kind they forward to in their ctt_type member
(which makes conceptual sense if you squint at it). The addition
machinery uses this to promote forwards to the appropriate type as
needed.
Unfortunately ctf_add_type does not: it checks the global namespace
(which is always wrong), and so fails with a spurious conflict if you
have, say, a typedef and then a forward comes along with the same name,
even if it's a forward to something like a struct. (This was observed
with <libio.h>, which has "struct _IO_FILE;" and also
"typedef struct _IO_FILE _IO_FILE"). We should look at the recorded
type kind and look in the appropriate namespace. We should also,
when creating the forward in the new container, use that type kind,
rather than just defaulting to CTF_K_STRUCT and hoping that what
eventually comes along is a struct.
This bug is as old as the first implementation of ctf_add_type in
Solaris. But we also want a new feature for the linker, closely-related
and touching the same code so we add it here: not only do we want a
forward followed by a struct/union/enum to promote the forward, but
we want want a struct/union/enum followed by a forward to act as a NOP
and return the existing type, because when we're adding many files
in succession to a target link, there will often be already-promoted
forwards (in the shape of a struct/union/enum) that want to unify
with duplicate forwards coming from other object files.
v5: fix tabdamage.
libctf/
* ctf-create.c (ctf_add_type): Look up and use the forwarded-to
type kind. Allow forwards to unify with pre-existing structs/
unions/enums.
2019-08-03 07:46:01 +08:00
|
|
|
union, or enum (i.e. the definition of the previous forward decl).
|
2019-04-24 18:22:03 +08:00
|
|
|
|
libctf: teach ctf_add_type how forwards work
This machinery has been broken for as long as Solaris has existed.
Forwards are meant to encode "struct foo;", "enum foo;" or "union
foo;". Obviously these all exist in distinct namespaces, so forwards
store the type kind they forward to in their ctt_type member
(which makes conceptual sense if you squint at it). The addition
machinery uses this to promote forwards to the appropriate type as
needed.
Unfortunately ctf_add_type does not: it checks the global namespace
(which is always wrong), and so fails with a spurious conflict if you
have, say, a typedef and then a forward comes along with the same name,
even if it's a forward to something like a struct. (This was observed
with <libio.h>, which has "struct _IO_FILE;" and also
"typedef struct _IO_FILE _IO_FILE"). We should look at the recorded
type kind and look in the appropriate namespace. We should also,
when creating the forward in the new container, use that type kind,
rather than just defaulting to CTF_K_STRUCT and hoping that what
eventually comes along is a struct.
This bug is as old as the first implementation of ctf_add_type in
Solaris. But we also want a new feature for the linker, closely-related
and touching the same code so we add it here: not only do we want a
forward followed by a struct/union/enum to promote the forward, but
we want want a struct/union/enum followed by a forward to act as a NOP
and return the existing type, because when we're adding many files
in succession to a target link, there will often be already-promoted
forwards (in the shape of a struct/union/enum) that want to unify
with duplicate forwards coming from other object files.
v5: fix tabdamage.
libctf/
* ctf-create.c (ctf_add_type): Look up and use the forwarded-to
type kind. Allow forwards to unify with pre-existing structs/
unions/enums.
2019-08-03 07:46:01 +08:00
|
|
|
We also allow addition in the opposite order (addition of a forward when a
|
|
|
|
struct, union, or enum already exists), which is a NOP and returns the
|
|
|
|
already-present struct, union, or enum. */
|
|
|
|
|
|
|
|
if (dst_type != CTF_ERR && dst_kind != kind)
|
2019-04-24 18:22:03 +08:00
|
|
|
{
|
libctf: teach ctf_add_type how forwards work
This machinery has been broken for as long as Solaris has existed.
Forwards are meant to encode "struct foo;", "enum foo;" or "union
foo;". Obviously these all exist in distinct namespaces, so forwards
store the type kind they forward to in their ctt_type member
(which makes conceptual sense if you squint at it). The addition
machinery uses this to promote forwards to the appropriate type as
needed.
Unfortunately ctf_add_type does not: it checks the global namespace
(which is always wrong), and so fails with a spurious conflict if you
have, say, a typedef and then a forward comes along with the same name,
even if it's a forward to something like a struct. (This was observed
with <libio.h>, which has "struct _IO_FILE;" and also
"typedef struct _IO_FILE _IO_FILE"). We should look at the recorded
type kind and look in the appropriate namespace. We should also,
when creating the forward in the new container, use that type kind,
rather than just defaulting to CTF_K_STRUCT and hoping that what
eventually comes along is a struct.
This bug is as old as the first implementation of ctf_add_type in
Solaris. But we also want a new feature for the linker, closely-related
and touching the same code so we add it here: not only do we want a
forward followed by a struct/union/enum to promote the forward, but
we want want a struct/union/enum followed by a forward to act as a NOP
and return the existing type, because when we're adding many files
in succession to a target link, there will often be already-promoted
forwards (in the shape of a struct/union/enum) that want to unify
with duplicate forwards coming from other object files.
v5: fix tabdamage.
libctf/
* ctf-create.c (ctf_add_type): Look up and use the forwarded-to
type kind. Allow forwards to unify with pre-existing structs/
unions/enums.
2019-08-03 07:46:01 +08:00
|
|
|
if (kind == CTF_K_FORWARD
|
|
|
|
&& (dst_kind == CTF_K_ENUM || dst_kind == CTF_K_STRUCT
|
|
|
|
|| dst_kind == CTF_K_UNION))
|
|
|
|
{
|
|
|
|
ctf_add_type_mapping (src_fp, src_type, dst_fp, dst_type);
|
|
|
|
return dst_type;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (dst_kind != CTF_K_FORWARD
|
|
|
|
|| (kind != CTF_K_ENUM && kind != CTF_K_STRUCT
|
|
|
|
&& kind != CTF_K_UNION))
|
|
|
|
{
|
|
|
|
ctf_dprintf ("Conflict for type %s: kinds differ, new: %i; "
|
|
|
|
"old (ID %lx): %i\n", name, kind, dst_type, dst_kind);
|
|
|
|
return (ctf_set_errno (dst_fp, ECTF_CONFLICT));
|
|
|
|
}
|
2019-04-24 18:22:03 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/* We take special action for an integer, float, or slice since it is
|
|
|
|
described not only by its name but also its encoding. For integers,
|
|
|
|
bit-fields exploit this degeneracy. */
|
|
|
|
|
|
|
|
if (kind == CTF_K_INTEGER || kind == CTF_K_FLOAT || kind == CTF_K_SLICE)
|
|
|
|
{
|
|
|
|
if (ctf_type_encoding (src_fp, src_type, &src_en) != 0)
|
|
|
|
return (ctf_set_errno (dst_fp, ctf_errno (src_fp)));
|
|
|
|
|
|
|
|
if (dst_type != CTF_ERR)
|
|
|
|
{
|
|
|
|
ctf_file_t *fp = dst_fp;
|
|
|
|
|
|
|
|
if ((dst_tp = ctf_lookup_by_id (&fp, dst_type)) == NULL)
|
|
|
|
return CTF_ERR;
|
|
|
|
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
if (ctf_type_encoding (dst_fp, dst_type, &dst_en) != 0)
|
|
|
|
return CTF_ERR; /* errno set for us. */
|
|
|
|
|
2019-04-24 18:22:03 +08:00
|
|
|
if (LCTF_INFO_ISROOT (fp, dst_tp->ctt_info) & CTF_ADD_ROOT)
|
|
|
|
{
|
|
|
|
/* The type that we found in the hash is also root-visible. If
|
|
|
|
the two types match then use the existing one; otherwise,
|
|
|
|
declare a conflict. Note: slices are not certain to match
|
|
|
|
even if there is no conflict: we must check the contained type
|
|
|
|
too. */
|
|
|
|
|
|
|
|
if (memcmp (&src_en, &dst_en, sizeof (ctf_encoding_t)) == 0)
|
|
|
|
{
|
|
|
|
if (kind != CTF_K_SLICE)
|
2019-07-14 04:31:26 +08:00
|
|
|
{
|
|
|
|
ctf_add_type_mapping (src_fp, src_type, dst_fp, dst_type);
|
|
|
|
return dst_type;
|
|
|
|
}
|
2019-04-24 18:22:03 +08:00
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
return (ctf_set_errno (dst_fp, ECTF_CONFLICT));
|
|
|
|
}
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
/* We found a non-root-visible type in the hash. If its encoding
|
|
|
|
is the same, we can reuse it, unless it is a slice. */
|
2019-04-24 18:22:03 +08:00
|
|
|
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
if (memcmp (&src_en, &dst_en, sizeof (ctf_encoding_t)) == 0)
|
2019-04-24 18:22:03 +08:00
|
|
|
{
|
|
|
|
if (kind != CTF_K_SLICE)
|
2019-07-14 04:31:26 +08:00
|
|
|
{
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
ctf_add_type_mapping (src_fp, src_type, dst_fp, dst_type);
|
|
|
|
return dst_type;
|
2019-07-14 04:31:26 +08:00
|
|
|
}
|
2019-04-24 18:22:03 +08:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
src.ctb_file = src_fp;
|
|
|
|
src.ctb_type = src_type;
|
|
|
|
src.ctb_dtd = NULL;
|
|
|
|
|
|
|
|
dst.ctb_file = dst_fp;
|
|
|
|
dst.ctb_type = dst_type;
|
|
|
|
dst.ctb_dtd = NULL;
|
|
|
|
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
/* Now perform kind-specific processing. If dst_type is CTF_ERR, then we add
|
|
|
|
a new type with the same properties as src_type to dst_fp. If dst_type is
|
|
|
|
not CTF_ERR, then we verify that dst_type has the same attributes as
|
|
|
|
src_type. We recurse for embedded references. Before we start, we note
|
|
|
|
that we are processing this type, to prevent infinite recursion: we do not
|
|
|
|
re-process any type that appears in this list. The list is emptied
|
|
|
|
wholesale at the end of processing everything in this recursive stack. */
|
|
|
|
|
|
|
|
if (ctf_dynhash_insert (proc_tracking_fp->ctf_add_processing,
|
|
|
|
(void *) (uintptr_t) src_type, (void *) 1) < 0)
|
|
|
|
return ctf_set_errno (dst_fp, ENOMEM);
|
|
|
|
|
2019-04-24 18:22:03 +08:00
|
|
|
switch (kind)
|
|
|
|
{
|
|
|
|
case CTF_K_INTEGER:
|
|
|
|
/* If we found a match we will have either returned it or declared a
|
|
|
|
conflict. */
|
|
|
|
dst_type = ctf_add_integer (dst_fp, flag, name, &src_en);
|
|
|
|
break;
|
|
|
|
|
|
|
|
case CTF_K_FLOAT:
|
|
|
|
/* If we found a match we will have either returned it or declared a
|
|
|
|
conflict. */
|
|
|
|
dst_type = ctf_add_float (dst_fp, flag, name, &src_en);
|
|
|
|
break;
|
|
|
|
|
|
|
|
case CTF_K_SLICE:
|
|
|
|
/* We have checked for conflicting encodings: now try to add the
|
|
|
|
contained type. */
|
|
|
|
src_type = ctf_type_reference (src_fp, src_type);
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
src_type = ctf_add_type_internal (dst_fp, src_fp, src_type,
|
|
|
|
proc_tracking_fp);
|
2019-04-24 18:22:03 +08:00
|
|
|
|
|
|
|
if (src_type == CTF_ERR)
|
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
|
|
|
|
dst_type = ctf_add_slice (dst_fp, flag, src_type, &src_en);
|
|
|
|
break;
|
|
|
|
|
|
|
|
case CTF_K_POINTER:
|
|
|
|
case CTF_K_VOLATILE:
|
|
|
|
case CTF_K_CONST:
|
|
|
|
case CTF_K_RESTRICT:
|
|
|
|
src_type = ctf_type_reference (src_fp, src_type);
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
src_type = ctf_add_type_internal (dst_fp, src_fp, src_type,
|
|
|
|
proc_tracking_fp);
|
2019-04-24 18:22:03 +08:00
|
|
|
|
|
|
|
if (src_type == CTF_ERR)
|
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
|
|
|
|
dst_type = ctf_add_reftype (dst_fp, flag, src_type, kind);
|
|
|
|
break;
|
|
|
|
|
|
|
|
case CTF_K_ARRAY:
|
libctf: fix a number of build problems found on Solaris and NetBSD
- Use of nonportable <endian.h>
- Use of qsort_r
- Use of zlib without appropriate magic to pull in the binutils zlib
- Use of off64_t without checking (fixed by dropping the unused fields
that need off64_t entirely)
- signedness problems due to long being too short a type on 32-bit
platforms: ctf_id_t is now 'unsigned long', and CTF_ERR must be
used only for functions that return ctf_id_t
- One lingering use of bzero() and of <sys/errno.h>
All fixed, using code from gnulib where possible.
Relatedly, set cts_size in a couple of places it was missed
(string table and symbol table loading upon ctf_bfdopen()).
binutils/
* objdump.c (make_ctfsect): Drop cts_type, cts_flags, and
cts_offset.
* readelf.c (shdr_to_ctf_sect): Likewise.
include/
* ctf-api.h (ctf_sect_t): Drop cts_type, cts_flags, and cts_offset.
(ctf_id_t): This is now an unsigned type.
(CTF_ERR): Cast it to ctf_id_t. Note that it should only be used
for ctf_id_t-returning functions.
libctf/
* Makefile.am (ZLIB): New.
(ZLIBINC): Likewise.
(AM_CFLAGS): Use them.
(libctf_a_LIBADD): New, for LIBOBJS.
* configure.ac: Check for zlib, endian.h, and qsort_r.
* ctf-endian.h: New, providing htole64 and le64toh.
* swap.h: Code style fixes.
(bswap_identity_64): New.
* qsort_r.c: New, from gnulib (with one added #include).
* ctf-decls.h: New, providing a conditional qsort_r declaration,
and unconditional definitions of MIN and MAX.
* ctf-impl.h: Use it. Do not use <sys/errno.h>.
(ctf_set_errno): Now returns unsigned long.
* ctf-util.c (ctf_set_errno): Adjust here too.
* ctf-archive.c: Use ctf-endian.h.
(ctf_arc_open_by_offset): Use memset, not bzero. Drop cts_type,
cts_flags and cts_offset.
(ctf_arc_write): Drop debugging dependent on the size of off_t.
* ctf-create.c: Provide a definition of roundup if not defined.
(ctf_create): Drop cts_type, cts_flags and cts_offset.
(ctf_add_reftype): Do not check if type IDs are below zero.
(ctf_add_slice): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_member_offset): Cast error-returning ssize_t's to size_t
when known error-free. Drop CTF_ERR usage for functions returning
int.
(ctf_add_member_encoded): Drop CTF_ERR usage for functions returning
int.
(ctf_add_variable): Likewise.
(enumcmp): Likewise.
(enumadd): Likewise.
(membcmp): Likewise.
(ctf_add_type): Likewise. Cast error-returning ssize_t's to size_t
when known error-free.
* ctf-dump.c (ctf_is_slice): Drop CTF_ERR usage for functions
returning int: use CTF_ERR for functions returning ctf_type_id.
(ctf_dump_label): Likewise.
(ctf_dump_objts): Likewise.
* ctf-labels.c (ctf_label_topmost): Likewise.
(ctf_label_iter): Likewise.
(ctf_label_info): Likewise.
* ctf-lookup.c (ctf_func_args): Likewise.
* ctf-open.c (upgrade_types): Cast to size_t where appropriate.
(ctf_bufopen): Likewise. Use zlib types as needed.
* ctf-types.c (ctf_member_iter): Drop CTF_ERR usage for functions
returning int.
(ctf_enum_iter): Likewise.
(ctf_type_size): Likewise.
(ctf_type_align): Likewise. Cast to size_t where appropriate.
(ctf_type_kind_unsliced): Likewise.
(ctf_type_kind): Likewise.
(ctf_type_encoding): Likewise.
(ctf_member_info): Likewise.
(ctf_array_info): Likewise.
(ctf_enum_value): Likewise.
(ctf_type_rvisit): Likewise.
* ctf-open-bfd.c (ctf_bfdopen): Drop cts_type, cts_flags and
cts_offset.
(ctf_simple_open): Likewise.
(ctf_bfdopen_ctfsect): Likewise. Set cts_size properly.
* Makefile.in: Regenerate.
* aclocal.m4: Likewise.
* config.h: Likewise.
* configure: Likewise.
2019-05-31 17:10:51 +08:00
|
|
|
if (ctf_array_info (src_fp, src_type, &src_ar) != 0)
|
2019-04-24 18:22:03 +08:00
|
|
|
return (ctf_set_errno (dst_fp, ctf_errno (src_fp)));
|
|
|
|
|
|
|
|
src_ar.ctr_contents =
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
ctf_add_type_internal (dst_fp, src_fp, src_ar.ctr_contents,
|
|
|
|
proc_tracking_fp);
|
|
|
|
src_ar.ctr_index = ctf_add_type_internal (dst_fp, src_fp,
|
|
|
|
src_ar.ctr_index,
|
|
|
|
proc_tracking_fp);
|
2019-04-24 18:22:03 +08:00
|
|
|
src_ar.ctr_nelems = src_ar.ctr_nelems;
|
|
|
|
|
|
|
|
if (src_ar.ctr_contents == CTF_ERR || src_ar.ctr_index == CTF_ERR)
|
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
|
|
|
|
if (dst_type != CTF_ERR)
|
|
|
|
{
|
|
|
|
if (ctf_array_info (dst_fp, dst_type, &dst_ar) != 0)
|
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
|
|
|
|
if (memcmp (&src_ar, &dst_ar, sizeof (ctf_arinfo_t)))
|
|
|
|
{
|
|
|
|
ctf_dprintf ("Conflict for type %s against ID %lx: "
|
|
|
|
"array info differs, old %lx/%lx/%x; "
|
|
|
|
"new: %lx/%lx/%x\n", name, dst_type,
|
|
|
|
src_ar.ctr_contents, src_ar.ctr_index,
|
|
|
|
src_ar.ctr_nelems, dst_ar.ctr_contents,
|
|
|
|
dst_ar.ctr_index, dst_ar.ctr_nelems);
|
|
|
|
return (ctf_set_errno (dst_fp, ECTF_CONFLICT));
|
|
|
|
}
|
|
|
|
}
|
|
|
|
else
|
|
|
|
dst_type = ctf_add_array (dst_fp, flag, &src_ar);
|
|
|
|
break;
|
|
|
|
|
|
|
|
case CTF_K_FUNCTION:
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
ctc.ctc_return = ctf_add_type_internal (dst_fp, src_fp,
|
|
|
|
src_tp->ctt_type,
|
|
|
|
proc_tracking_fp);
|
2019-04-24 18:22:03 +08:00
|
|
|
ctc.ctc_argc = 0;
|
|
|
|
ctc.ctc_flags = 0;
|
|
|
|
|
|
|
|
if (ctc.ctc_return == CTF_ERR)
|
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
|
|
|
|
dst_type = ctf_add_function (dst_fp, flag, &ctc, NULL);
|
|
|
|
break;
|
|
|
|
|
|
|
|
case CTF_K_STRUCT:
|
|
|
|
case CTF_K_UNION:
|
|
|
|
{
|
|
|
|
ctf_dmdef_t *dmd;
|
|
|
|
int errs = 0;
|
libctf: fix a number of build problems found on Solaris and NetBSD
- Use of nonportable <endian.h>
- Use of qsort_r
- Use of zlib without appropriate magic to pull in the binutils zlib
- Use of off64_t without checking (fixed by dropping the unused fields
that need off64_t entirely)
- signedness problems due to long being too short a type on 32-bit
platforms: ctf_id_t is now 'unsigned long', and CTF_ERR must be
used only for functions that return ctf_id_t
- One lingering use of bzero() and of <sys/errno.h>
All fixed, using code from gnulib where possible.
Relatedly, set cts_size in a couple of places it was missed
(string table and symbol table loading upon ctf_bfdopen()).
binutils/
* objdump.c (make_ctfsect): Drop cts_type, cts_flags, and
cts_offset.
* readelf.c (shdr_to_ctf_sect): Likewise.
include/
* ctf-api.h (ctf_sect_t): Drop cts_type, cts_flags, and cts_offset.
(ctf_id_t): This is now an unsigned type.
(CTF_ERR): Cast it to ctf_id_t. Note that it should only be used
for ctf_id_t-returning functions.
libctf/
* Makefile.am (ZLIB): New.
(ZLIBINC): Likewise.
(AM_CFLAGS): Use them.
(libctf_a_LIBADD): New, for LIBOBJS.
* configure.ac: Check for zlib, endian.h, and qsort_r.
* ctf-endian.h: New, providing htole64 and le64toh.
* swap.h: Code style fixes.
(bswap_identity_64): New.
* qsort_r.c: New, from gnulib (with one added #include).
* ctf-decls.h: New, providing a conditional qsort_r declaration,
and unconditional definitions of MIN and MAX.
* ctf-impl.h: Use it. Do not use <sys/errno.h>.
(ctf_set_errno): Now returns unsigned long.
* ctf-util.c (ctf_set_errno): Adjust here too.
* ctf-archive.c: Use ctf-endian.h.
(ctf_arc_open_by_offset): Use memset, not bzero. Drop cts_type,
cts_flags and cts_offset.
(ctf_arc_write): Drop debugging dependent on the size of off_t.
* ctf-create.c: Provide a definition of roundup if not defined.
(ctf_create): Drop cts_type, cts_flags and cts_offset.
(ctf_add_reftype): Do not check if type IDs are below zero.
(ctf_add_slice): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_member_offset): Cast error-returning ssize_t's to size_t
when known error-free. Drop CTF_ERR usage for functions returning
int.
(ctf_add_member_encoded): Drop CTF_ERR usage for functions returning
int.
(ctf_add_variable): Likewise.
(enumcmp): Likewise.
(enumadd): Likewise.
(membcmp): Likewise.
(ctf_add_type): Likewise. Cast error-returning ssize_t's to size_t
when known error-free.
* ctf-dump.c (ctf_is_slice): Drop CTF_ERR usage for functions
returning int: use CTF_ERR for functions returning ctf_type_id.
(ctf_dump_label): Likewise.
(ctf_dump_objts): Likewise.
* ctf-labels.c (ctf_label_topmost): Likewise.
(ctf_label_iter): Likewise.
(ctf_label_info): Likewise.
* ctf-lookup.c (ctf_func_args): Likewise.
* ctf-open.c (upgrade_types): Cast to size_t where appropriate.
(ctf_bufopen): Likewise. Use zlib types as needed.
* ctf-types.c (ctf_member_iter): Drop CTF_ERR usage for functions
returning int.
(ctf_enum_iter): Likewise.
(ctf_type_size): Likewise.
(ctf_type_align): Likewise. Cast to size_t where appropriate.
(ctf_type_kind_unsliced): Likewise.
(ctf_type_kind): Likewise.
(ctf_type_encoding): Likewise.
(ctf_member_info): Likewise.
(ctf_array_info): Likewise.
(ctf_enum_value): Likewise.
(ctf_type_rvisit): Likewise.
* ctf-open-bfd.c (ctf_bfdopen): Drop cts_type, cts_flags and
cts_offset.
(ctf_simple_open): Likewise.
(ctf_bfdopen_ctfsect): Likewise. Set cts_size properly.
* Makefile.in: Regenerate.
* aclocal.m4: Likewise.
* config.h: Likewise.
* configure: Likewise.
2019-05-31 17:10:51 +08:00
|
|
|
size_t size;
|
|
|
|
ssize_t ssize;
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
ctf_dtdef_t *dtd;
|
2019-04-24 18:22:03 +08:00
|
|
|
|
|
|
|
/* Technically to match a struct or union we need to check both
|
|
|
|
ways (src members vs. dst, dst members vs. src) but we make
|
|
|
|
this more optimal by only checking src vs. dst and comparing
|
|
|
|
the total size of the structure (which we must do anyway)
|
|
|
|
which covers the possibility of dst members not in src.
|
|
|
|
This optimization can be defeated for unions, but is so
|
|
|
|
pathological as to render it irrelevant for our purposes. */
|
|
|
|
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
if (dst_type != CTF_ERR && kind != CTF_K_FORWARD
|
|
|
|
&& dst_kind != CTF_K_FORWARD)
|
2019-04-24 18:22:03 +08:00
|
|
|
{
|
|
|
|
if (ctf_type_size (src_fp, src_type) !=
|
|
|
|
ctf_type_size (dst_fp, dst_type))
|
|
|
|
{
|
|
|
|
ctf_dprintf ("Conflict for type %s against ID %lx: "
|
2019-06-05 20:34:36 +08:00
|
|
|
"union size differs, old %li, new %li\n",
|
|
|
|
name, dst_type,
|
|
|
|
(long) ctf_type_size (src_fp, src_type),
|
|
|
|
(long) ctf_type_size (dst_fp, dst_type));
|
2019-04-24 18:22:03 +08:00
|
|
|
return (ctf_set_errno (dst_fp, ECTF_CONFLICT));
|
|
|
|
}
|
|
|
|
|
|
|
|
if (ctf_member_iter (src_fp, src_type, membcmp, &dst))
|
|
|
|
{
|
|
|
|
ctf_dprintf ("Conflict for type %s against ID %lx: "
|
|
|
|
"members differ, see above\n", name, dst_type);
|
|
|
|
return (ctf_set_errno (dst_fp, ECTF_CONFLICT));
|
|
|
|
}
|
|
|
|
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Unlike the other cases, copying structs and unions is done
|
|
|
|
manually so as to avoid repeated lookups in ctf_add_member
|
|
|
|
and to ensure the exact same member offsets as in src_type. */
|
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
dst_type = ctf_add_generic (dst_fp, flag, name, kind, &dtd);
|
2019-04-24 18:22:03 +08:00
|
|
|
if (dst_type == CTF_ERR)
|
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
|
|
|
|
dst.ctb_type = dst_type;
|
|
|
|
dst.ctb_dtd = dtd;
|
|
|
|
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
/* Pre-emptively add this struct to the type mapping so that
|
|
|
|
structures that refer to themselves work. */
|
|
|
|
ctf_add_type_mapping (src_fp, src_type, dst_fp, dst_type);
|
|
|
|
|
2019-04-24 18:22:03 +08:00
|
|
|
if (ctf_member_iter (src_fp, src_type, membadd, &dst) != 0)
|
|
|
|
errs++; /* Increment errs and fail at bottom of case. */
|
|
|
|
|
libctf: fix a number of build problems found on Solaris and NetBSD
- Use of nonportable <endian.h>
- Use of qsort_r
- Use of zlib without appropriate magic to pull in the binutils zlib
- Use of off64_t without checking (fixed by dropping the unused fields
that need off64_t entirely)
- signedness problems due to long being too short a type on 32-bit
platforms: ctf_id_t is now 'unsigned long', and CTF_ERR must be
used only for functions that return ctf_id_t
- One lingering use of bzero() and of <sys/errno.h>
All fixed, using code from gnulib where possible.
Relatedly, set cts_size in a couple of places it was missed
(string table and symbol table loading upon ctf_bfdopen()).
binutils/
* objdump.c (make_ctfsect): Drop cts_type, cts_flags, and
cts_offset.
* readelf.c (shdr_to_ctf_sect): Likewise.
include/
* ctf-api.h (ctf_sect_t): Drop cts_type, cts_flags, and cts_offset.
(ctf_id_t): This is now an unsigned type.
(CTF_ERR): Cast it to ctf_id_t. Note that it should only be used
for ctf_id_t-returning functions.
libctf/
* Makefile.am (ZLIB): New.
(ZLIBINC): Likewise.
(AM_CFLAGS): Use them.
(libctf_a_LIBADD): New, for LIBOBJS.
* configure.ac: Check for zlib, endian.h, and qsort_r.
* ctf-endian.h: New, providing htole64 and le64toh.
* swap.h: Code style fixes.
(bswap_identity_64): New.
* qsort_r.c: New, from gnulib (with one added #include).
* ctf-decls.h: New, providing a conditional qsort_r declaration,
and unconditional definitions of MIN and MAX.
* ctf-impl.h: Use it. Do not use <sys/errno.h>.
(ctf_set_errno): Now returns unsigned long.
* ctf-util.c (ctf_set_errno): Adjust here too.
* ctf-archive.c: Use ctf-endian.h.
(ctf_arc_open_by_offset): Use memset, not bzero. Drop cts_type,
cts_flags and cts_offset.
(ctf_arc_write): Drop debugging dependent on the size of off_t.
* ctf-create.c: Provide a definition of roundup if not defined.
(ctf_create): Drop cts_type, cts_flags and cts_offset.
(ctf_add_reftype): Do not check if type IDs are below zero.
(ctf_add_slice): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_member_offset): Cast error-returning ssize_t's to size_t
when known error-free. Drop CTF_ERR usage for functions returning
int.
(ctf_add_member_encoded): Drop CTF_ERR usage for functions returning
int.
(ctf_add_variable): Likewise.
(enumcmp): Likewise.
(enumadd): Likewise.
(membcmp): Likewise.
(ctf_add_type): Likewise. Cast error-returning ssize_t's to size_t
when known error-free.
* ctf-dump.c (ctf_is_slice): Drop CTF_ERR usage for functions
returning int: use CTF_ERR for functions returning ctf_type_id.
(ctf_dump_label): Likewise.
(ctf_dump_objts): Likewise.
* ctf-labels.c (ctf_label_topmost): Likewise.
(ctf_label_iter): Likewise.
(ctf_label_info): Likewise.
* ctf-lookup.c (ctf_func_args): Likewise.
* ctf-open.c (upgrade_types): Cast to size_t where appropriate.
(ctf_bufopen): Likewise. Use zlib types as needed.
* ctf-types.c (ctf_member_iter): Drop CTF_ERR usage for functions
returning int.
(ctf_enum_iter): Likewise.
(ctf_type_size): Likewise.
(ctf_type_align): Likewise. Cast to size_t where appropriate.
(ctf_type_kind_unsliced): Likewise.
(ctf_type_kind): Likewise.
(ctf_type_encoding): Likewise.
(ctf_member_info): Likewise.
(ctf_array_info): Likewise.
(ctf_enum_value): Likewise.
(ctf_type_rvisit): Likewise.
* ctf-open-bfd.c (ctf_bfdopen): Drop cts_type, cts_flags and
cts_offset.
(ctf_simple_open): Likewise.
(ctf_bfdopen_ctfsect): Likewise. Set cts_size properly.
* Makefile.in: Regenerate.
* aclocal.m4: Likewise.
* config.h: Likewise.
* configure: Likewise.
2019-05-31 17:10:51 +08:00
|
|
|
if ((ssize = ctf_type_size (src_fp, src_type)) < 0)
|
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
|
|
|
|
size = (size_t) ssize;
|
|
|
|
if (size > CTF_MAX_SIZE)
|
2019-04-24 18:22:03 +08:00
|
|
|
{
|
|
|
|
dtd->dtd_data.ctt_size = CTF_LSIZE_SENT;
|
|
|
|
dtd->dtd_data.ctt_lsizehi = CTF_SIZE_TO_LSIZE_HI (size);
|
|
|
|
dtd->dtd_data.ctt_lsizelo = CTF_SIZE_TO_LSIZE_LO (size);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
dtd->dtd_data.ctt_size = (uint32_t) size;
|
|
|
|
|
|
|
|
dtd->dtd_data.ctt_info = CTF_TYPE_INFO (kind, flag, vlen);
|
|
|
|
|
|
|
|
/* Make a final pass through the members changing each dmd_type (a
|
|
|
|
src_fp type) to an equivalent type in dst_fp. We pass through all
|
libctf: handle nonrepresentable types at link time
GCC can emit references to type 0 to indicate that this type is one that
is not representable in the version of CTF it emits (for instance,
version 3 cannot encode vector types). Type 0 is already used in the
function section to indicate padding inserted to skip functions we do
not want to encode the type of, so using zero in this way is a good
extension of the format: but libctf reports such types as ECTF_BADID,
which is indistinguishable from file corruption via links to truly
nonexistent types with IDs like 0xDEADBEEF etc, which we really do want
to stop for.
In particular, this stops all traversals of types dead at this point,
preventing us from even dumping CTF files containing unrepresentable
types to see what's going on!
So add a new error, ECTF_NONREPRESENTABLE, which is returned by
recursive type resolution when a reference to a zero type is found. (No
zero type is ever emitted into the CTF file by GCC, only references to
one). We can't do much with types that are ultimately nonrepresentable,
but we can do enough to keep functioning.
Adjust ctf_add_type to ensure that top-level types of type zero and
structure and union members of ultimate type zero are simply skipped
without reporting an error, so we can copy structures and unions that
contain nonrepresentable members (skipping them and leaving a hole where
they would be, so no consumers downstream of the linker need to worry
about this): adjust the dumper so that we dump members of
nonrepresentable types in a simple form that indicates
nonrepresentability rather than terminating the dump, and do not falsely
assume all errors to be -ENOMEM: adjust the linker so that types that
fail to get added are simply skipped, so that both nonrepresentable
types and outright errors do not terminate the type addition, which
could skip many valid types and cause further errors when variables of
those types are added.
In future, when we gain the ability to call back to the linker to report
link-time type resolution errors, we should report failures to add all
but nonrepresentable types. But we can't do that yet.
v5: Fix tabdamage.
include/
* ctf-api.h (ECTF_NONREPRESENTABLE): New.
libctf/
* ctf-types.c (ctf_type_resolve): Return ECTF_NONREPRESENTABLE on
type zero.
* ctf-create.c (ctf_add_type): Detect and skip nonrepresentable
members and types.
(ctf_add_variable): Likewise for variables pointing to them.
* ctf-link.c (ctf_link_one_type): Do not warn for nonrepresentable
type link failure, but do warn for others.
* ctf-dump.c (ctf_dump_format_type): Likewise. Do not assume all
errors to be ENOMEM.
(ctf_dump_member): Likewise.
(ctf_dump_type): Likewise.
(ctf_dump_header_strfield): Do not assume all errors to be ENOMEM.
(ctf_dump_header_sectfield): Do not assume all errors to be ENOMEM.
(ctf_dump_header): Likewise.
(ctf_dump_label): likewise.
(ctf_dump_objts): likewise.
(ctf_dump_funcs): likewise.
(ctf_dump_var): likewise.
(ctf_dump_str): Likewise.
2019-08-05 18:40:33 +08:00
|
|
|
members, leaving any that fail set to CTF_ERR, unless they fail
|
|
|
|
because they are marking a member of type not representable in this
|
|
|
|
version of CTF, in which case we just want to silently omit them:
|
|
|
|
no consumer can do anything with them anyway. */
|
2019-04-24 18:22:03 +08:00
|
|
|
for (dmd = ctf_list_next (&dtd->dtd_u.dtu_members);
|
|
|
|
dmd != NULL; dmd = ctf_list_next (dmd))
|
|
|
|
{
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
ctf_file_t *dst = dst_fp;
|
|
|
|
ctf_id_t memb_type;
|
|
|
|
|
|
|
|
memb_type = ctf_type_mapping (src_fp, dmd->dmd_type, &dst);
|
|
|
|
if (memb_type == 0)
|
libctf: handle nonrepresentable types at link time
GCC can emit references to type 0 to indicate that this type is one that
is not representable in the version of CTF it emits (for instance,
version 3 cannot encode vector types). Type 0 is already used in the
function section to indicate padding inserted to skip functions we do
not want to encode the type of, so using zero in this way is a good
extension of the format: but libctf reports such types as ECTF_BADID,
which is indistinguishable from file corruption via links to truly
nonexistent types with IDs like 0xDEADBEEF etc, which we really do want
to stop for.
In particular, this stops all traversals of types dead at this point,
preventing us from even dumping CTF files containing unrepresentable
types to see what's going on!
So add a new error, ECTF_NONREPRESENTABLE, which is returned by
recursive type resolution when a reference to a zero type is found. (No
zero type is ever emitted into the CTF file by GCC, only references to
one). We can't do much with types that are ultimately nonrepresentable,
but we can do enough to keep functioning.
Adjust ctf_add_type to ensure that top-level types of type zero and
structure and union members of ultimate type zero are simply skipped
without reporting an error, so we can copy structures and unions that
contain nonrepresentable members (skipping them and leaving a hole where
they would be, so no consumers downstream of the linker need to worry
about this): adjust the dumper so that we dump members of
nonrepresentable types in a simple form that indicates
nonrepresentability rather than terminating the dump, and do not falsely
assume all errors to be -ENOMEM: adjust the linker so that types that
fail to get added are simply skipped, so that both nonrepresentable
types and outright errors do not terminate the type addition, which
could skip many valid types and cause further errors when variables of
those types are added.
In future, when we gain the ability to call back to the linker to report
link-time type resolution errors, we should report failures to add all
but nonrepresentable types. But we can't do that yet.
v5: Fix tabdamage.
include/
* ctf-api.h (ECTF_NONREPRESENTABLE): New.
libctf/
* ctf-types.c (ctf_type_resolve): Return ECTF_NONREPRESENTABLE on
type zero.
* ctf-create.c (ctf_add_type): Detect and skip nonrepresentable
members and types.
(ctf_add_variable): Likewise for variables pointing to them.
* ctf-link.c (ctf_link_one_type): Do not warn for nonrepresentable
type link failure, but do warn for others.
* ctf-dump.c (ctf_dump_format_type): Likewise. Do not assume all
errors to be ENOMEM.
(ctf_dump_member): Likewise.
(ctf_dump_type): Likewise.
(ctf_dump_header_strfield): Do not assume all errors to be ENOMEM.
(ctf_dump_header_sectfield): Do not assume all errors to be ENOMEM.
(ctf_dump_header): Likewise.
(ctf_dump_label): likewise.
(ctf_dump_objts): likewise.
(ctf_dump_funcs): likewise.
(ctf_dump_var): likewise.
(ctf_dump_str): Likewise.
2019-08-05 18:40:33 +08:00
|
|
|
{
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
if ((dmd->dmd_type =
|
|
|
|
ctf_add_type_internal (dst_fp, src_fp, dmd->dmd_type,
|
|
|
|
proc_tracking_fp)) == CTF_ERR)
|
|
|
|
{
|
|
|
|
if (ctf_errno (dst_fp) != ECTF_NONREPRESENTABLE)
|
|
|
|
errs++;
|
|
|
|
}
|
libctf: handle nonrepresentable types at link time
GCC can emit references to type 0 to indicate that this type is one that
is not representable in the version of CTF it emits (for instance,
version 3 cannot encode vector types). Type 0 is already used in the
function section to indicate padding inserted to skip functions we do
not want to encode the type of, so using zero in this way is a good
extension of the format: but libctf reports such types as ECTF_BADID,
which is indistinguishable from file corruption via links to truly
nonexistent types with IDs like 0xDEADBEEF etc, which we really do want
to stop for.
In particular, this stops all traversals of types dead at this point,
preventing us from even dumping CTF files containing unrepresentable
types to see what's going on!
So add a new error, ECTF_NONREPRESENTABLE, which is returned by
recursive type resolution when a reference to a zero type is found. (No
zero type is ever emitted into the CTF file by GCC, only references to
one). We can't do much with types that are ultimately nonrepresentable,
but we can do enough to keep functioning.
Adjust ctf_add_type to ensure that top-level types of type zero and
structure and union members of ultimate type zero are simply skipped
without reporting an error, so we can copy structures and unions that
contain nonrepresentable members (skipping them and leaving a hole where
they would be, so no consumers downstream of the linker need to worry
about this): adjust the dumper so that we dump members of
nonrepresentable types in a simple form that indicates
nonrepresentability rather than terminating the dump, and do not falsely
assume all errors to be -ENOMEM: adjust the linker so that types that
fail to get added are simply skipped, so that both nonrepresentable
types and outright errors do not terminate the type addition, which
could skip many valid types and cause further errors when variables of
those types are added.
In future, when we gain the ability to call back to the linker to report
link-time type resolution errors, we should report failures to add all
but nonrepresentable types. But we can't do that yet.
v5: Fix tabdamage.
include/
* ctf-api.h (ECTF_NONREPRESENTABLE): New.
libctf/
* ctf-types.c (ctf_type_resolve): Return ECTF_NONREPRESENTABLE on
type zero.
* ctf-create.c (ctf_add_type): Detect and skip nonrepresentable
members and types.
(ctf_add_variable): Likewise for variables pointing to them.
* ctf-link.c (ctf_link_one_type): Do not warn for nonrepresentable
type link failure, but do warn for others.
* ctf-dump.c (ctf_dump_format_type): Likewise. Do not assume all
errors to be ENOMEM.
(ctf_dump_member): Likewise.
(ctf_dump_type): Likewise.
(ctf_dump_header_strfield): Do not assume all errors to be ENOMEM.
(ctf_dump_header_sectfield): Do not assume all errors to be ENOMEM.
(ctf_dump_header): Likewise.
(ctf_dump_label): likewise.
(ctf_dump_objts): likewise.
(ctf_dump_funcs): likewise.
(ctf_dump_var): likewise.
(ctf_dump_str): Likewise.
2019-08-05 18:40:33 +08:00
|
|
|
}
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
else
|
|
|
|
dmd->dmd_type = memb_type;
|
2019-04-24 18:22:03 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
if (errs)
|
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
case CTF_K_ENUM:
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
if (dst_type != CTF_ERR && kind != CTF_K_FORWARD
|
|
|
|
&& dst_kind != CTF_K_FORWARD)
|
2019-04-24 18:22:03 +08:00
|
|
|
{
|
|
|
|
if (ctf_enum_iter (src_fp, src_type, enumcmp, &dst)
|
|
|
|
|| ctf_enum_iter (dst_fp, dst_type, enumcmp, &src))
|
|
|
|
{
|
|
|
|
ctf_dprintf ("Conflict for enum %s against ID %lx: "
|
|
|
|
"members differ, see above\n", name, dst_type);
|
|
|
|
return (ctf_set_errno (dst_fp, ECTF_CONFLICT));
|
|
|
|
}
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
dst_type = ctf_add_enum (dst_fp, flag, name);
|
|
|
|
if ((dst.ctb_type = dst_type) == CTF_ERR
|
|
|
|
|| ctf_enum_iter (src_fp, src_type, enumadd, &dst))
|
|
|
|
return CTF_ERR; /* errno is set for us */
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
|
|
|
|
case CTF_K_FORWARD:
|
|
|
|
if (dst_type == CTF_ERR)
|
libctf: teach ctf_add_type how forwards work
This machinery has been broken for as long as Solaris has existed.
Forwards are meant to encode "struct foo;", "enum foo;" or "union
foo;". Obviously these all exist in distinct namespaces, so forwards
store the type kind they forward to in their ctt_type member
(which makes conceptual sense if you squint at it). The addition
machinery uses this to promote forwards to the appropriate type as
needed.
Unfortunately ctf_add_type does not: it checks the global namespace
(which is always wrong), and so fails with a spurious conflict if you
have, say, a typedef and then a forward comes along with the same name,
even if it's a forward to something like a struct. (This was observed
with <libio.h>, which has "struct _IO_FILE;" and also
"typedef struct _IO_FILE _IO_FILE"). We should look at the recorded
type kind and look in the appropriate namespace. We should also,
when creating the forward in the new container, use that type kind,
rather than just defaulting to CTF_K_STRUCT and hoping that what
eventually comes along is a struct.
This bug is as old as the first implementation of ctf_add_type in
Solaris. But we also want a new feature for the linker, closely-related
and touching the same code so we add it here: not only do we want a
forward followed by a struct/union/enum to promote the forward, but
we want want a struct/union/enum followed by a forward to act as a NOP
and return the existing type, because when we're adding many files
in succession to a target link, there will often be already-promoted
forwards (in the shape of a struct/union/enum) that want to unify
with duplicate forwards coming from other object files.
v5: fix tabdamage.
libctf/
* ctf-create.c (ctf_add_type): Look up and use the forwarded-to
type kind. Allow forwards to unify with pre-existing structs/
unions/enums.
2019-08-03 07:46:01 +08:00
|
|
|
dst_type = ctf_add_forward (dst_fp, flag, name, forward_kind);
|
2019-04-24 18:22:03 +08:00
|
|
|
break;
|
|
|
|
|
|
|
|
case CTF_K_TYPEDEF:
|
|
|
|
src_type = ctf_type_reference (src_fp, src_type);
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
src_type = ctf_add_type_internal (dst_fp, src_fp, src_type,
|
|
|
|
proc_tracking_fp);
|
2019-04-24 18:22:03 +08:00
|
|
|
|
|
|
|
if (src_type == CTF_ERR)
|
|
|
|
return CTF_ERR; /* errno is set for us. */
|
|
|
|
|
|
|
|
/* If dst_type is not CTF_ERR at this point, we should check if
|
|
|
|
ctf_type_reference(dst_fp, dst_type) != src_type and if so fail with
|
|
|
|
ECTF_CONFLICT. However, this causes problems with bitness typedefs
|
|
|
|
that vary based on things like if 32-bit then pid_t is int otherwise
|
|
|
|
long. We therefore omit this check and assume that if the identically
|
|
|
|
named typedef already exists in dst_fp, it is correct or
|
|
|
|
equivalent. */
|
|
|
|
|
|
|
|
if (dst_type == CTF_ERR)
|
|
|
|
dst_type = ctf_add_typedef (dst_fp, flag, name, src_type);
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
|
2019-04-24 18:22:03 +08:00
|
|
|
break;
|
|
|
|
|
|
|
|
default:
|
|
|
|
return (ctf_set_errno (dst_fp, ECTF_CORRUPT));
|
|
|
|
}
|
|
|
|
|
2019-07-14 04:31:26 +08:00
|
|
|
if (dst_type != CTF_ERR)
|
|
|
|
ctf_add_type_mapping (src_fp, orig_src_type, dst_fp, dst_type);
|
2019-04-24 18:22:03 +08:00
|
|
|
return dst_type;
|
|
|
|
}
|
|
|
|
|
libctf: properly handle ctf_add_type of forwards and self-reffing structs
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
2019-08-08 01:01:08 +08:00
|
|
|
ctf_id_t
|
|
|
|
ctf_add_type (ctf_file_t *dst_fp, ctf_file_t *src_fp, ctf_id_t src_type)
|
|
|
|
{
|
|
|
|
ctf_id_t id;
|
|
|
|
|
|
|
|
if (!src_fp->ctf_add_processing)
|
|
|
|
src_fp->ctf_add_processing = ctf_dynhash_create (ctf_hash_integer,
|
|
|
|
ctf_hash_eq_integer,
|
|
|
|
NULL, NULL);
|
|
|
|
|
|
|
|
/* We store the hash on the source, because it contains only source type IDs:
|
|
|
|
but callers will invariably expect errors to appear on the dest. */
|
|
|
|
if (!src_fp->ctf_add_processing)
|
|
|
|
return (ctf_set_errno (dst_fp, ENOMEM));
|
|
|
|
|
|
|
|
id = ctf_add_type_internal (dst_fp, src_fp, src_type, src_fp);
|
|
|
|
ctf_dynhash_empty (src_fp->ctf_add_processing);
|
|
|
|
|
|
|
|
return id;
|
|
|
|
}
|
|
|
|
|
libctf: allow the header to change between versions
libctf supports dynamic upgrading of the type table as file format
versions change, but before now has not supported changes to the CTF
header. Doing this is complicated by the baroque storage method used:
the CTF header is kept prepended to the rest of the CTF data, just as
when read from the file, and written out from there, and is
endian-flipped in place.
This makes accessing it needlessly hard and makes it almost impossible
to make the header larger if we add fields. The general storage
machinery around the malloced ctf pointer (the 'ctf_base') is also
overcomplicated: the pointer is sometimes malloced locally and sometimes
assigned from a parameter, so freeing it requires checking to see if
that parameter was used, needlessly coupling ctf_bufopen and
ctf_file_close together.
So split the header out into a new ctf_file_t.ctf_header, which is
written out explicitly: squeeze it out of the CTF buffer whenever we
reallocate it, and use ctf_file_t.ctf_buf to skip past the header when
we do not need to reallocate (when no upgrading or endian-flipping is
required). We now track whether the CTF base can be freed explicitly
via a new ctf_dynbase pointer which is non-NULL only when freeing is
possible.
With all this done, we can upgrade the header on the fly and add new
fields as desired, via a new upgrade_header function in ctf-open.
As with other forms of upgrading, libctf upgrades older headers
automatically to the latest supported version at open time.
For a first use of this field, we add a new string field cth_cuname, and
a corresponding setter/getter pair ctf_cuname_set and ctf_cuname: this
is used by debuggers to determine whether a CTF section's types relate
to a single compilation unit, or to all compilation units in the
program. (Types with ambiguous definitions in different CUs have only
one of these types placed in the top-level shared .ctf container: the
rest are placed in much smaller per-CU containers, which have the shared
container as their parent. Since CTF must be useful in the absence of
DWARF, we store the names of the relevant CUs ourselves, so the debugger
can look them up.)
v5: fix tabdamage.
include/
* ctf-api.h (ctf_cuname): New function.
(ctf_cuname_set): Likewise.
* ctf.h: Improve comment around upgrading, no longer
implying that v2 is the target of upgrades (it is v3 now).
(ctf_header_v2_t): New, old-format header for backward
compatibility.
(ctf_header_t): Add cth_cuname: this is the first of several
header changes in format v3.
libctf/
* ctf-impl.h (ctf_file_t): New fields ctf_header, ctf_dynbase,
ctf_cuname, ctf_dyncuname: ctf_base and ctf_buf are no longer const.
* ctf-open.c (ctf_set_base): Preserve the gap between ctf_buf and
ctf_base: do not assume that it is always sizeof (ctf_header_t).
Print out ctf_cuname: only print out ctf_parname if set.
(ctf_free_base): Removed, ctf_base is no longer freed: free
ctf_dynbase instead.
(ctf_set_version): Fix spacing.
(upgrade_header): New, in-place header upgrading.
(upgrade_types): Rename to...
(upgrade_types_v1): ... this. Free ctf_dynbase, not ctf_base. No
longer track old and new headers separately. No longer allow for
header sizes explicitly: squeeze the headers out on upgrade (they
are preserved in fp->ctf_header). Set ctf_dynbase, ctf_base and
ctf_buf explicitly. Use ctf_free, not ctf_free_base.
(upgrade_types): New, also handle ctf_parmax updating.
(flip_header): Flip ctf_cuname.
(flip_types): Flip BUF explicitly rather than deriving BUF from
BASE.
(ctf_bufopen): Store the header in fp->ctf_header. Correct minimum
required alignment of objtoff and funcoff. No longer store it in
the ctf_buf unless that buf is derived unmodified from the input.
Set ctf_dynbase where ctf_base is dynamically allocated. Drop locals
that duplicate fields in ctf_file: move allocation of ctf_file
further up instead. Call upgrade_header as needed. Move
version-specific ctf_parmax initialization into upgrade_types. More
concise error handling.
(ctf_file_close): No longer test for null pointers before freeing.
Free ctf_dyncuname, ctf_dynbase, and ctf_header. Do not call
ctf_free_base.
(ctf_cuname): New.
(ctf_cuname_set): New.
* ctf-create.c (ctf_update): Populate ctf_cuname.
(ctf_gzwrite): Write out the header explicitly. Remove obsolescent
comment.
(ctf_write): Likewise.
(ctf_compress_write): Get the header from ctf_header, not ctf_base.
Fix the compression length: fp->ctf_size never counted the CTF
header. Simplify the compress call accordingly.
2019-07-07 00:36:21 +08:00
|
|
|
/* Write the compressed CTF data stream to the specified gzFile descriptor. */
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
int
|
|
|
|
ctf_gzwrite (ctf_file_t *fp, gzFile fd)
|
|
|
|
{
|
libctf: allow the header to change between versions
libctf supports dynamic upgrading of the type table as file format
versions change, but before now has not supported changes to the CTF
header. Doing this is complicated by the baroque storage method used:
the CTF header is kept prepended to the rest of the CTF data, just as
when read from the file, and written out from there, and is
endian-flipped in place.
This makes accessing it needlessly hard and makes it almost impossible
to make the header larger if we add fields. The general storage
machinery around the malloced ctf pointer (the 'ctf_base') is also
overcomplicated: the pointer is sometimes malloced locally and sometimes
assigned from a parameter, so freeing it requires checking to see if
that parameter was used, needlessly coupling ctf_bufopen and
ctf_file_close together.
So split the header out into a new ctf_file_t.ctf_header, which is
written out explicitly: squeeze it out of the CTF buffer whenever we
reallocate it, and use ctf_file_t.ctf_buf to skip past the header when
we do not need to reallocate (when no upgrading or endian-flipping is
required). We now track whether the CTF base can be freed explicitly
via a new ctf_dynbase pointer which is non-NULL only when freeing is
possible.
With all this done, we can upgrade the header on the fly and add new
fields as desired, via a new upgrade_header function in ctf-open.
As with other forms of upgrading, libctf upgrades older headers
automatically to the latest supported version at open time.
For a first use of this field, we add a new string field cth_cuname, and
a corresponding setter/getter pair ctf_cuname_set and ctf_cuname: this
is used by debuggers to determine whether a CTF section's types relate
to a single compilation unit, or to all compilation units in the
program. (Types with ambiguous definitions in different CUs have only
one of these types placed in the top-level shared .ctf container: the
rest are placed in much smaller per-CU containers, which have the shared
container as their parent. Since CTF must be useful in the absence of
DWARF, we store the names of the relevant CUs ourselves, so the debugger
can look them up.)
v5: fix tabdamage.
include/
* ctf-api.h (ctf_cuname): New function.
(ctf_cuname_set): Likewise.
* ctf.h: Improve comment around upgrading, no longer
implying that v2 is the target of upgrades (it is v3 now).
(ctf_header_v2_t): New, old-format header for backward
compatibility.
(ctf_header_t): Add cth_cuname: this is the first of several
header changes in format v3.
libctf/
* ctf-impl.h (ctf_file_t): New fields ctf_header, ctf_dynbase,
ctf_cuname, ctf_dyncuname: ctf_base and ctf_buf are no longer const.
* ctf-open.c (ctf_set_base): Preserve the gap between ctf_buf and
ctf_base: do not assume that it is always sizeof (ctf_header_t).
Print out ctf_cuname: only print out ctf_parname if set.
(ctf_free_base): Removed, ctf_base is no longer freed: free
ctf_dynbase instead.
(ctf_set_version): Fix spacing.
(upgrade_header): New, in-place header upgrading.
(upgrade_types): Rename to...
(upgrade_types_v1): ... this. Free ctf_dynbase, not ctf_base. No
longer track old and new headers separately. No longer allow for
header sizes explicitly: squeeze the headers out on upgrade (they
are preserved in fp->ctf_header). Set ctf_dynbase, ctf_base and
ctf_buf explicitly. Use ctf_free, not ctf_free_base.
(upgrade_types): New, also handle ctf_parmax updating.
(flip_header): Flip ctf_cuname.
(flip_types): Flip BUF explicitly rather than deriving BUF from
BASE.
(ctf_bufopen): Store the header in fp->ctf_header. Correct minimum
required alignment of objtoff and funcoff. No longer store it in
the ctf_buf unless that buf is derived unmodified from the input.
Set ctf_dynbase where ctf_base is dynamically allocated. Drop locals
that duplicate fields in ctf_file: move allocation of ctf_file
further up instead. Call upgrade_header as needed. Move
version-specific ctf_parmax initialization into upgrade_types. More
concise error handling.
(ctf_file_close): No longer test for null pointers before freeing.
Free ctf_dyncuname, ctf_dynbase, and ctf_header. Do not call
ctf_free_base.
(ctf_cuname): New.
(ctf_cuname_set): New.
* ctf-create.c (ctf_update): Populate ctf_cuname.
(ctf_gzwrite): Write out the header explicitly. Remove obsolescent
comment.
(ctf_write): Likewise.
(ctf_compress_write): Get the header from ctf_header, not ctf_base.
Fix the compression length: fp->ctf_size never counted the CTF
header. Simplify the compress call accordingly.
2019-07-07 00:36:21 +08:00
|
|
|
const unsigned char *buf;
|
|
|
|
ssize_t resid;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
ssize_t len;
|
|
|
|
|
libctf: allow the header to change between versions
libctf supports dynamic upgrading of the type table as file format
versions change, but before now has not supported changes to the CTF
header. Doing this is complicated by the baroque storage method used:
the CTF header is kept prepended to the rest of the CTF data, just as
when read from the file, and written out from there, and is
endian-flipped in place.
This makes accessing it needlessly hard and makes it almost impossible
to make the header larger if we add fields. The general storage
machinery around the malloced ctf pointer (the 'ctf_base') is also
overcomplicated: the pointer is sometimes malloced locally and sometimes
assigned from a parameter, so freeing it requires checking to see if
that parameter was used, needlessly coupling ctf_bufopen and
ctf_file_close together.
So split the header out into a new ctf_file_t.ctf_header, which is
written out explicitly: squeeze it out of the CTF buffer whenever we
reallocate it, and use ctf_file_t.ctf_buf to skip past the header when
we do not need to reallocate (when no upgrading or endian-flipping is
required). We now track whether the CTF base can be freed explicitly
via a new ctf_dynbase pointer which is non-NULL only when freeing is
possible.
With all this done, we can upgrade the header on the fly and add new
fields as desired, via a new upgrade_header function in ctf-open.
As with other forms of upgrading, libctf upgrades older headers
automatically to the latest supported version at open time.
For a first use of this field, we add a new string field cth_cuname, and
a corresponding setter/getter pair ctf_cuname_set and ctf_cuname: this
is used by debuggers to determine whether a CTF section's types relate
to a single compilation unit, or to all compilation units in the
program. (Types with ambiguous definitions in different CUs have only
one of these types placed in the top-level shared .ctf container: the
rest are placed in much smaller per-CU containers, which have the shared
container as their parent. Since CTF must be useful in the absence of
DWARF, we store the names of the relevant CUs ourselves, so the debugger
can look them up.)
v5: fix tabdamage.
include/
* ctf-api.h (ctf_cuname): New function.
(ctf_cuname_set): Likewise.
* ctf.h: Improve comment around upgrading, no longer
implying that v2 is the target of upgrades (it is v3 now).
(ctf_header_v2_t): New, old-format header for backward
compatibility.
(ctf_header_t): Add cth_cuname: this is the first of several
header changes in format v3.
libctf/
* ctf-impl.h (ctf_file_t): New fields ctf_header, ctf_dynbase,
ctf_cuname, ctf_dyncuname: ctf_base and ctf_buf are no longer const.
* ctf-open.c (ctf_set_base): Preserve the gap between ctf_buf and
ctf_base: do not assume that it is always sizeof (ctf_header_t).
Print out ctf_cuname: only print out ctf_parname if set.
(ctf_free_base): Removed, ctf_base is no longer freed: free
ctf_dynbase instead.
(ctf_set_version): Fix spacing.
(upgrade_header): New, in-place header upgrading.
(upgrade_types): Rename to...
(upgrade_types_v1): ... this. Free ctf_dynbase, not ctf_base. No
longer track old and new headers separately. No longer allow for
header sizes explicitly: squeeze the headers out on upgrade (they
are preserved in fp->ctf_header). Set ctf_dynbase, ctf_base and
ctf_buf explicitly. Use ctf_free, not ctf_free_base.
(upgrade_types): New, also handle ctf_parmax updating.
(flip_header): Flip ctf_cuname.
(flip_types): Flip BUF explicitly rather than deriving BUF from
BASE.
(ctf_bufopen): Store the header in fp->ctf_header. Correct minimum
required alignment of objtoff and funcoff. No longer store it in
the ctf_buf unless that buf is derived unmodified from the input.
Set ctf_dynbase where ctf_base is dynamically allocated. Drop locals
that duplicate fields in ctf_file: move allocation of ctf_file
further up instead. Call upgrade_header as needed. Move
version-specific ctf_parmax initialization into upgrade_types. More
concise error handling.
(ctf_file_close): No longer test for null pointers before freeing.
Free ctf_dyncuname, ctf_dynbase, and ctf_header. Do not call
ctf_free_base.
(ctf_cuname): New.
(ctf_cuname_set): New.
* ctf-create.c (ctf_update): Populate ctf_cuname.
(ctf_gzwrite): Write out the header explicitly. Remove obsolescent
comment.
(ctf_write): Likewise.
(ctf_compress_write): Get the header from ctf_header, not ctf_base.
Fix the compression length: fp->ctf_size never counted the CTF
header. Simplify the compress call accordingly.
2019-07-07 00:36:21 +08:00
|
|
|
resid = sizeof (ctf_header_t);
|
|
|
|
buf = (unsigned char *) fp->ctf_header;
|
|
|
|
while (resid != 0)
|
|
|
|
{
|
|
|
|
if ((len = gzwrite (fd, buf, resid)) <= 0)
|
|
|
|
return (ctf_set_errno (fp, errno));
|
|
|
|
resid -= len;
|
|
|
|
buf += len;
|
|
|
|
}
|
|
|
|
|
|
|
|
resid = fp->ctf_size;
|
|
|
|
buf = fp->ctf_buf;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
while (resid != 0)
|
|
|
|
{
|
|
|
|
if ((len = gzwrite (fd, buf, resid)) <= 0)
|
|
|
|
return (ctf_set_errno (fp, errno));
|
|
|
|
resid -= len;
|
|
|
|
buf += len;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Compress the specified CTF data stream and write it to the specified file
|
|
|
|
descriptor. */
|
|
|
|
int
|
|
|
|
ctf_compress_write (ctf_file_t *fp, int fd)
|
|
|
|
{
|
|
|
|
unsigned char *buf;
|
|
|
|
unsigned char *bp;
|
|
|
|
ctf_header_t h;
|
|
|
|
ctf_header_t *hp = &h;
|
|
|
|
ssize_t header_len = sizeof (ctf_header_t);
|
|
|
|
ssize_t compress_len;
|
|
|
|
ssize_t len;
|
|
|
|
int rc;
|
|
|
|
int err = 0;
|
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
if (ctf_serialize (fp) < 0)
|
|
|
|
return -1; /* errno is set for us. */
|
|
|
|
|
libctf: allow the header to change between versions
libctf supports dynamic upgrading of the type table as file format
versions change, but before now has not supported changes to the CTF
header. Doing this is complicated by the baroque storage method used:
the CTF header is kept prepended to the rest of the CTF data, just as
when read from the file, and written out from there, and is
endian-flipped in place.
This makes accessing it needlessly hard and makes it almost impossible
to make the header larger if we add fields. The general storage
machinery around the malloced ctf pointer (the 'ctf_base') is also
overcomplicated: the pointer is sometimes malloced locally and sometimes
assigned from a parameter, so freeing it requires checking to see if
that parameter was used, needlessly coupling ctf_bufopen and
ctf_file_close together.
So split the header out into a new ctf_file_t.ctf_header, which is
written out explicitly: squeeze it out of the CTF buffer whenever we
reallocate it, and use ctf_file_t.ctf_buf to skip past the header when
we do not need to reallocate (when no upgrading or endian-flipping is
required). We now track whether the CTF base can be freed explicitly
via a new ctf_dynbase pointer which is non-NULL only when freeing is
possible.
With all this done, we can upgrade the header on the fly and add new
fields as desired, via a new upgrade_header function in ctf-open.
As with other forms of upgrading, libctf upgrades older headers
automatically to the latest supported version at open time.
For a first use of this field, we add a new string field cth_cuname, and
a corresponding setter/getter pair ctf_cuname_set and ctf_cuname: this
is used by debuggers to determine whether a CTF section's types relate
to a single compilation unit, or to all compilation units in the
program. (Types with ambiguous definitions in different CUs have only
one of these types placed in the top-level shared .ctf container: the
rest are placed in much smaller per-CU containers, which have the shared
container as their parent. Since CTF must be useful in the absence of
DWARF, we store the names of the relevant CUs ourselves, so the debugger
can look them up.)
v5: fix tabdamage.
include/
* ctf-api.h (ctf_cuname): New function.
(ctf_cuname_set): Likewise.
* ctf.h: Improve comment around upgrading, no longer
implying that v2 is the target of upgrades (it is v3 now).
(ctf_header_v2_t): New, old-format header for backward
compatibility.
(ctf_header_t): Add cth_cuname: this is the first of several
header changes in format v3.
libctf/
* ctf-impl.h (ctf_file_t): New fields ctf_header, ctf_dynbase,
ctf_cuname, ctf_dyncuname: ctf_base and ctf_buf are no longer const.
* ctf-open.c (ctf_set_base): Preserve the gap between ctf_buf and
ctf_base: do not assume that it is always sizeof (ctf_header_t).
Print out ctf_cuname: only print out ctf_parname if set.
(ctf_free_base): Removed, ctf_base is no longer freed: free
ctf_dynbase instead.
(ctf_set_version): Fix spacing.
(upgrade_header): New, in-place header upgrading.
(upgrade_types): Rename to...
(upgrade_types_v1): ... this. Free ctf_dynbase, not ctf_base. No
longer track old and new headers separately. No longer allow for
header sizes explicitly: squeeze the headers out on upgrade (they
are preserved in fp->ctf_header). Set ctf_dynbase, ctf_base and
ctf_buf explicitly. Use ctf_free, not ctf_free_base.
(upgrade_types): New, also handle ctf_parmax updating.
(flip_header): Flip ctf_cuname.
(flip_types): Flip BUF explicitly rather than deriving BUF from
BASE.
(ctf_bufopen): Store the header in fp->ctf_header. Correct minimum
required alignment of objtoff and funcoff. No longer store it in
the ctf_buf unless that buf is derived unmodified from the input.
Set ctf_dynbase where ctf_base is dynamically allocated. Drop locals
that duplicate fields in ctf_file: move allocation of ctf_file
further up instead. Call upgrade_header as needed. Move
version-specific ctf_parmax initialization into upgrade_types. More
concise error handling.
(ctf_file_close): No longer test for null pointers before freeing.
Free ctf_dyncuname, ctf_dynbase, and ctf_header. Do not call
ctf_free_base.
(ctf_cuname): New.
(ctf_cuname_set): New.
* ctf-create.c (ctf_update): Populate ctf_cuname.
(ctf_gzwrite): Write out the header explicitly. Remove obsolescent
comment.
(ctf_write): Likewise.
(ctf_compress_write): Get the header from ctf_header, not ctf_base.
Fix the compression length: fp->ctf_size never counted the CTF
header. Simplify the compress call accordingly.
2019-07-07 00:36:21 +08:00
|
|
|
memcpy (hp, fp->ctf_header, header_len);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
hp->cth_flags |= CTF_F_COMPRESS;
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
compress_len = compressBound (fp->ctf_size);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
if ((buf = malloc (compress_len)) == NULL)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return (ctf_set_errno (fp, ECTF_ZALLOC));
|
|
|
|
|
libctf: drop mmap()-based CTF data allocator
This allocator has the ostensible benefit that it lets us mprotect() the
memory used for CTF storage: but in exchange for this it adds
considerable complexity, since we have to track allocation sizes
ourselves for use at freeing time, note whether the data we are storing
was ctf_data_alloc()ed or not so we know if we can safely mprotect()
it... and while the mprotect()ing has found few bugs, it *has* been the
cause of more than one due to errors in all this tracking leading to us
mprotect()ing bits of the heap and stuff like that.
We are about to start composing CTF buffers from pieces so that we can
do usage-based optimizations on the strtab. This means we need
realloc(), which needs nonportable mremap() and *more* tracking of the
*original* allocation size, and the complexity and bureaucracy of all of
this is just too high for its negligible benefits.
Drop the whole thing and just use malloc() like everyone else. It knows
better than we do when it is safe to use mmap() under the covers,
anyway.
While we're at it, don't leak the entire buffer if ctf_compress_write()
fails to compress it.
libctf/
* ctf-subr.c (_PAGESIZE): Remove.
(ctf_data_alloc): Likewise.
(ctf_data_free): Likewise.
(ctf_data_protect): Likewise.
* ctf-impl.h: Remove declarations.
* ctf-create.c (ctf_update): No longer call ctf_data_protect: use
ctf_free, not ctf_data_free.
(ctf_compress_write): Use ctf_data_alloc, not ctf_alloc. Free
the buffer again on compression error.
* ctf-open.c (ctf_set_base): No longer track the size: call
ctf_free, not ctf_data_free.
(upgrade_types): Likewise. Call ctf_alloc, not ctf_data_alloc.
(ctf_bufopen): Likewise. No longer call ctf_data_protect.
2019-06-19 19:20:47 +08:00
|
|
|
if ((rc = compress (buf, (uLongf *) &compress_len,
|
libctf: allow the header to change between versions
libctf supports dynamic upgrading of the type table as file format
versions change, but before now has not supported changes to the CTF
header. Doing this is complicated by the baroque storage method used:
the CTF header is kept prepended to the rest of the CTF data, just as
when read from the file, and written out from there, and is
endian-flipped in place.
This makes accessing it needlessly hard and makes it almost impossible
to make the header larger if we add fields. The general storage
machinery around the malloced ctf pointer (the 'ctf_base') is also
overcomplicated: the pointer is sometimes malloced locally and sometimes
assigned from a parameter, so freeing it requires checking to see if
that parameter was used, needlessly coupling ctf_bufopen and
ctf_file_close together.
So split the header out into a new ctf_file_t.ctf_header, which is
written out explicitly: squeeze it out of the CTF buffer whenever we
reallocate it, and use ctf_file_t.ctf_buf to skip past the header when
we do not need to reallocate (when no upgrading or endian-flipping is
required). We now track whether the CTF base can be freed explicitly
via a new ctf_dynbase pointer which is non-NULL only when freeing is
possible.
With all this done, we can upgrade the header on the fly and add new
fields as desired, via a new upgrade_header function in ctf-open.
As with other forms of upgrading, libctf upgrades older headers
automatically to the latest supported version at open time.
For a first use of this field, we add a new string field cth_cuname, and
a corresponding setter/getter pair ctf_cuname_set and ctf_cuname: this
is used by debuggers to determine whether a CTF section's types relate
to a single compilation unit, or to all compilation units in the
program. (Types with ambiguous definitions in different CUs have only
one of these types placed in the top-level shared .ctf container: the
rest are placed in much smaller per-CU containers, which have the shared
container as their parent. Since CTF must be useful in the absence of
DWARF, we store the names of the relevant CUs ourselves, so the debugger
can look them up.)
v5: fix tabdamage.
include/
* ctf-api.h (ctf_cuname): New function.
(ctf_cuname_set): Likewise.
* ctf.h: Improve comment around upgrading, no longer
implying that v2 is the target of upgrades (it is v3 now).
(ctf_header_v2_t): New, old-format header for backward
compatibility.
(ctf_header_t): Add cth_cuname: this is the first of several
header changes in format v3.
libctf/
* ctf-impl.h (ctf_file_t): New fields ctf_header, ctf_dynbase,
ctf_cuname, ctf_dyncuname: ctf_base and ctf_buf are no longer const.
* ctf-open.c (ctf_set_base): Preserve the gap between ctf_buf and
ctf_base: do not assume that it is always sizeof (ctf_header_t).
Print out ctf_cuname: only print out ctf_parname if set.
(ctf_free_base): Removed, ctf_base is no longer freed: free
ctf_dynbase instead.
(ctf_set_version): Fix spacing.
(upgrade_header): New, in-place header upgrading.
(upgrade_types): Rename to...
(upgrade_types_v1): ... this. Free ctf_dynbase, not ctf_base. No
longer track old and new headers separately. No longer allow for
header sizes explicitly: squeeze the headers out on upgrade (they
are preserved in fp->ctf_header). Set ctf_dynbase, ctf_base and
ctf_buf explicitly. Use ctf_free, not ctf_free_base.
(upgrade_types): New, also handle ctf_parmax updating.
(flip_header): Flip ctf_cuname.
(flip_types): Flip BUF explicitly rather than deriving BUF from
BASE.
(ctf_bufopen): Store the header in fp->ctf_header. Correct minimum
required alignment of objtoff and funcoff. No longer store it in
the ctf_buf unless that buf is derived unmodified from the input.
Set ctf_dynbase where ctf_base is dynamically allocated. Drop locals
that duplicate fields in ctf_file: move allocation of ctf_file
further up instead. Call upgrade_header as needed. Move
version-specific ctf_parmax initialization into upgrade_types. More
concise error handling.
(ctf_file_close): No longer test for null pointers before freeing.
Free ctf_dyncuname, ctf_dynbase, and ctf_header. Do not call
ctf_free_base.
(ctf_cuname): New.
(ctf_cuname_set): New.
* ctf-create.c (ctf_update): Populate ctf_cuname.
(ctf_gzwrite): Write out the header explicitly. Remove obsolescent
comment.
(ctf_write): Likewise.
(ctf_compress_write): Get the header from ctf_header, not ctf_base.
Fix the compression length: fp->ctf_size never counted the CTF
header. Simplify the compress call accordingly.
2019-07-07 00:36:21 +08:00
|
|
|
fp->ctf_buf, fp->ctf_size)) != Z_OK)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
{
|
|
|
|
ctf_dprintf ("zlib deflate err: %s\n", zError (rc));
|
|
|
|
err = ctf_set_errno (fp, ECTF_COMPRESS);
|
|
|
|
goto ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
while (header_len > 0)
|
|
|
|
{
|
|
|
|
if ((len = write (fd, hp, header_len)) < 0)
|
|
|
|
{
|
|
|
|
err = ctf_set_errno (fp, errno);
|
|
|
|
goto ret;
|
|
|
|
}
|
|
|
|
header_len -= len;
|
|
|
|
hp += len;
|
|
|
|
}
|
|
|
|
|
|
|
|
bp = buf;
|
|
|
|
while (compress_len > 0)
|
|
|
|
{
|
|
|
|
if ((len = write (fd, bp, compress_len)) < 0)
|
|
|
|
{
|
|
|
|
err = ctf_set_errno (fp, errno);
|
|
|
|
goto ret;
|
|
|
|
}
|
|
|
|
compress_len -= len;
|
|
|
|
bp += len;
|
|
|
|
}
|
|
|
|
|
|
|
|
ret:
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
free (buf);
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2019-07-14 03:40:52 +08:00
|
|
|
/* Optionally compress the specified CTF data stream and return it as a new
|
|
|
|
dynamically-allocated string. */
|
|
|
|
unsigned char *
|
|
|
|
ctf_write_mem (ctf_file_t *fp, size_t *size, size_t threshold)
|
|
|
|
{
|
|
|
|
unsigned char *buf;
|
|
|
|
unsigned char *bp;
|
|
|
|
ctf_header_t *hp;
|
|
|
|
ssize_t header_len = sizeof (ctf_header_t);
|
|
|
|
ssize_t compress_len;
|
|
|
|
int rc;
|
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
if (ctf_serialize (fp) < 0)
|
|
|
|
return NULL; /* errno is set for us. */
|
|
|
|
|
|
|
|
compress_len = compressBound (fp->ctf_size);
|
2019-07-14 03:40:52 +08:00
|
|
|
if (fp->ctf_size < threshold)
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
compress_len = fp->ctf_size;
|
|
|
|
if ((buf = malloc (compress_len
|
2019-07-14 03:40:52 +08:00
|
|
|
+ sizeof (struct ctf_header))) == NULL)
|
|
|
|
{
|
|
|
|
ctf_set_errno (fp, ENOMEM);
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
hp = (ctf_header_t *) buf;
|
|
|
|
memcpy (hp, fp->ctf_header, header_len);
|
|
|
|
bp = buf + sizeof (struct ctf_header);
|
|
|
|
*size = sizeof (struct ctf_header);
|
|
|
|
|
|
|
|
if (fp->ctf_size < threshold)
|
|
|
|
{
|
|
|
|
hp->cth_flags &= ~CTF_F_COMPRESS;
|
|
|
|
memcpy (bp, fp->ctf_buf, fp->ctf_size);
|
|
|
|
*size += fp->ctf_size;
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
hp->cth_flags |= CTF_F_COMPRESS;
|
|
|
|
if ((rc = compress (bp, (uLongf *) &compress_len,
|
|
|
|
fp->ctf_buf, fp->ctf_size)) != Z_OK)
|
|
|
|
{
|
|
|
|
ctf_dprintf ("zlib deflate err: %s\n", zError (rc));
|
|
|
|
ctf_set_errno (fp, ECTF_COMPRESS);
|
libctf: remove ctf_malloc, ctf_free and ctf_strdup
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
2019-09-17 13:54:23 +08:00
|
|
|
free (buf);
|
2019-07-14 03:40:52 +08:00
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
*size += compress_len;
|
|
|
|
}
|
|
|
|
return buf;
|
|
|
|
}
|
|
|
|
|
libctf: allow the header to change between versions
libctf supports dynamic upgrading of the type table as file format
versions change, but before now has not supported changes to the CTF
header. Doing this is complicated by the baroque storage method used:
the CTF header is kept prepended to the rest of the CTF data, just as
when read from the file, and written out from there, and is
endian-flipped in place.
This makes accessing it needlessly hard and makes it almost impossible
to make the header larger if we add fields. The general storage
machinery around the malloced ctf pointer (the 'ctf_base') is also
overcomplicated: the pointer is sometimes malloced locally and sometimes
assigned from a parameter, so freeing it requires checking to see if
that parameter was used, needlessly coupling ctf_bufopen and
ctf_file_close together.
So split the header out into a new ctf_file_t.ctf_header, which is
written out explicitly: squeeze it out of the CTF buffer whenever we
reallocate it, and use ctf_file_t.ctf_buf to skip past the header when
we do not need to reallocate (when no upgrading or endian-flipping is
required). We now track whether the CTF base can be freed explicitly
via a new ctf_dynbase pointer which is non-NULL only when freeing is
possible.
With all this done, we can upgrade the header on the fly and add new
fields as desired, via a new upgrade_header function in ctf-open.
As with other forms of upgrading, libctf upgrades older headers
automatically to the latest supported version at open time.
For a first use of this field, we add a new string field cth_cuname, and
a corresponding setter/getter pair ctf_cuname_set and ctf_cuname: this
is used by debuggers to determine whether a CTF section's types relate
to a single compilation unit, or to all compilation units in the
program. (Types with ambiguous definitions in different CUs have only
one of these types placed in the top-level shared .ctf container: the
rest are placed in much smaller per-CU containers, which have the shared
container as their parent. Since CTF must be useful in the absence of
DWARF, we store the names of the relevant CUs ourselves, so the debugger
can look them up.)
v5: fix tabdamage.
include/
* ctf-api.h (ctf_cuname): New function.
(ctf_cuname_set): Likewise.
* ctf.h: Improve comment around upgrading, no longer
implying that v2 is the target of upgrades (it is v3 now).
(ctf_header_v2_t): New, old-format header for backward
compatibility.
(ctf_header_t): Add cth_cuname: this is the first of several
header changes in format v3.
libctf/
* ctf-impl.h (ctf_file_t): New fields ctf_header, ctf_dynbase,
ctf_cuname, ctf_dyncuname: ctf_base and ctf_buf are no longer const.
* ctf-open.c (ctf_set_base): Preserve the gap between ctf_buf and
ctf_base: do not assume that it is always sizeof (ctf_header_t).
Print out ctf_cuname: only print out ctf_parname if set.
(ctf_free_base): Removed, ctf_base is no longer freed: free
ctf_dynbase instead.
(ctf_set_version): Fix spacing.
(upgrade_header): New, in-place header upgrading.
(upgrade_types): Rename to...
(upgrade_types_v1): ... this. Free ctf_dynbase, not ctf_base. No
longer track old and new headers separately. No longer allow for
header sizes explicitly: squeeze the headers out on upgrade (they
are preserved in fp->ctf_header). Set ctf_dynbase, ctf_base and
ctf_buf explicitly. Use ctf_free, not ctf_free_base.
(upgrade_types): New, also handle ctf_parmax updating.
(flip_header): Flip ctf_cuname.
(flip_types): Flip BUF explicitly rather than deriving BUF from
BASE.
(ctf_bufopen): Store the header in fp->ctf_header. Correct minimum
required alignment of objtoff and funcoff. No longer store it in
the ctf_buf unless that buf is derived unmodified from the input.
Set ctf_dynbase where ctf_base is dynamically allocated. Drop locals
that duplicate fields in ctf_file: move allocation of ctf_file
further up instead. Call upgrade_header as needed. Move
version-specific ctf_parmax initialization into upgrade_types. More
concise error handling.
(ctf_file_close): No longer test for null pointers before freeing.
Free ctf_dyncuname, ctf_dynbase, and ctf_header. Do not call
ctf_free_base.
(ctf_cuname): New.
(ctf_cuname_set): New.
* ctf-create.c (ctf_update): Populate ctf_cuname.
(ctf_gzwrite): Write out the header explicitly. Remove obsolescent
comment.
(ctf_write): Likewise.
(ctf_compress_write): Get the header from ctf_header, not ctf_base.
Fix the compression length: fp->ctf_size never counted the CTF
header. Simplify the compress call accordingly.
2019-07-07 00:36:21 +08:00
|
|
|
/* Write the uncompressed CTF data stream to the specified file descriptor. */
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
int
|
|
|
|
ctf_write (ctf_file_t *fp, int fd)
|
|
|
|
{
|
libctf: allow the header to change between versions
libctf supports dynamic upgrading of the type table as file format
versions change, but before now has not supported changes to the CTF
header. Doing this is complicated by the baroque storage method used:
the CTF header is kept prepended to the rest of the CTF data, just as
when read from the file, and written out from there, and is
endian-flipped in place.
This makes accessing it needlessly hard and makes it almost impossible
to make the header larger if we add fields. The general storage
machinery around the malloced ctf pointer (the 'ctf_base') is also
overcomplicated: the pointer is sometimes malloced locally and sometimes
assigned from a parameter, so freeing it requires checking to see if
that parameter was used, needlessly coupling ctf_bufopen and
ctf_file_close together.
So split the header out into a new ctf_file_t.ctf_header, which is
written out explicitly: squeeze it out of the CTF buffer whenever we
reallocate it, and use ctf_file_t.ctf_buf to skip past the header when
we do not need to reallocate (when no upgrading or endian-flipping is
required). We now track whether the CTF base can be freed explicitly
via a new ctf_dynbase pointer which is non-NULL only when freeing is
possible.
With all this done, we can upgrade the header on the fly and add new
fields as desired, via a new upgrade_header function in ctf-open.
As with other forms of upgrading, libctf upgrades older headers
automatically to the latest supported version at open time.
For a first use of this field, we add a new string field cth_cuname, and
a corresponding setter/getter pair ctf_cuname_set and ctf_cuname: this
is used by debuggers to determine whether a CTF section's types relate
to a single compilation unit, or to all compilation units in the
program. (Types with ambiguous definitions in different CUs have only
one of these types placed in the top-level shared .ctf container: the
rest are placed in much smaller per-CU containers, which have the shared
container as their parent. Since CTF must be useful in the absence of
DWARF, we store the names of the relevant CUs ourselves, so the debugger
can look them up.)
v5: fix tabdamage.
include/
* ctf-api.h (ctf_cuname): New function.
(ctf_cuname_set): Likewise.
* ctf.h: Improve comment around upgrading, no longer
implying that v2 is the target of upgrades (it is v3 now).
(ctf_header_v2_t): New, old-format header for backward
compatibility.
(ctf_header_t): Add cth_cuname: this is the first of several
header changes in format v3.
libctf/
* ctf-impl.h (ctf_file_t): New fields ctf_header, ctf_dynbase,
ctf_cuname, ctf_dyncuname: ctf_base and ctf_buf are no longer const.
* ctf-open.c (ctf_set_base): Preserve the gap between ctf_buf and
ctf_base: do not assume that it is always sizeof (ctf_header_t).
Print out ctf_cuname: only print out ctf_parname if set.
(ctf_free_base): Removed, ctf_base is no longer freed: free
ctf_dynbase instead.
(ctf_set_version): Fix spacing.
(upgrade_header): New, in-place header upgrading.
(upgrade_types): Rename to...
(upgrade_types_v1): ... this. Free ctf_dynbase, not ctf_base. No
longer track old and new headers separately. No longer allow for
header sizes explicitly: squeeze the headers out on upgrade (they
are preserved in fp->ctf_header). Set ctf_dynbase, ctf_base and
ctf_buf explicitly. Use ctf_free, not ctf_free_base.
(upgrade_types): New, also handle ctf_parmax updating.
(flip_header): Flip ctf_cuname.
(flip_types): Flip BUF explicitly rather than deriving BUF from
BASE.
(ctf_bufopen): Store the header in fp->ctf_header. Correct minimum
required alignment of objtoff and funcoff. No longer store it in
the ctf_buf unless that buf is derived unmodified from the input.
Set ctf_dynbase where ctf_base is dynamically allocated. Drop locals
that duplicate fields in ctf_file: move allocation of ctf_file
further up instead. Call upgrade_header as needed. Move
version-specific ctf_parmax initialization into upgrade_types. More
concise error handling.
(ctf_file_close): No longer test for null pointers before freeing.
Free ctf_dyncuname, ctf_dynbase, and ctf_header. Do not call
ctf_free_base.
(ctf_cuname): New.
(ctf_cuname_set): New.
* ctf-create.c (ctf_update): Populate ctf_cuname.
(ctf_gzwrite): Write out the header explicitly. Remove obsolescent
comment.
(ctf_write): Likewise.
(ctf_compress_write): Get the header from ctf_header, not ctf_base.
Fix the compression length: fp->ctf_size never counted the CTF
header. Simplify the compress call accordingly.
2019-07-07 00:36:21 +08:00
|
|
|
const unsigned char *buf;
|
|
|
|
ssize_t resid;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
ssize_t len;
|
|
|
|
|
libctf: avoid the need to ever use ctf_update
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
2019-08-08 00:55:09 +08:00
|
|
|
if (ctf_serialize (fp) < 0)
|
|
|
|
return -1; /* errno is set for us. */
|
|
|
|
|
libctf: allow the header to change between versions
libctf supports dynamic upgrading of the type table as file format
versions change, but before now has not supported changes to the CTF
header. Doing this is complicated by the baroque storage method used:
the CTF header is kept prepended to the rest of the CTF data, just as
when read from the file, and written out from there, and is
endian-flipped in place.
This makes accessing it needlessly hard and makes it almost impossible
to make the header larger if we add fields. The general storage
machinery around the malloced ctf pointer (the 'ctf_base') is also
overcomplicated: the pointer is sometimes malloced locally and sometimes
assigned from a parameter, so freeing it requires checking to see if
that parameter was used, needlessly coupling ctf_bufopen and
ctf_file_close together.
So split the header out into a new ctf_file_t.ctf_header, which is
written out explicitly: squeeze it out of the CTF buffer whenever we
reallocate it, and use ctf_file_t.ctf_buf to skip past the header when
we do not need to reallocate (when no upgrading or endian-flipping is
required). We now track whether the CTF base can be freed explicitly
via a new ctf_dynbase pointer which is non-NULL only when freeing is
possible.
With all this done, we can upgrade the header on the fly and add new
fields as desired, via a new upgrade_header function in ctf-open.
As with other forms of upgrading, libctf upgrades older headers
automatically to the latest supported version at open time.
For a first use of this field, we add a new string field cth_cuname, and
a corresponding setter/getter pair ctf_cuname_set and ctf_cuname: this
is used by debuggers to determine whether a CTF section's types relate
to a single compilation unit, or to all compilation units in the
program. (Types with ambiguous definitions in different CUs have only
one of these types placed in the top-level shared .ctf container: the
rest are placed in much smaller per-CU containers, which have the shared
container as their parent. Since CTF must be useful in the absence of
DWARF, we store the names of the relevant CUs ourselves, so the debugger
can look them up.)
v5: fix tabdamage.
include/
* ctf-api.h (ctf_cuname): New function.
(ctf_cuname_set): Likewise.
* ctf.h: Improve comment around upgrading, no longer
implying that v2 is the target of upgrades (it is v3 now).
(ctf_header_v2_t): New, old-format header for backward
compatibility.
(ctf_header_t): Add cth_cuname: this is the first of several
header changes in format v3.
libctf/
* ctf-impl.h (ctf_file_t): New fields ctf_header, ctf_dynbase,
ctf_cuname, ctf_dyncuname: ctf_base and ctf_buf are no longer const.
* ctf-open.c (ctf_set_base): Preserve the gap between ctf_buf and
ctf_base: do not assume that it is always sizeof (ctf_header_t).
Print out ctf_cuname: only print out ctf_parname if set.
(ctf_free_base): Removed, ctf_base is no longer freed: free
ctf_dynbase instead.
(ctf_set_version): Fix spacing.
(upgrade_header): New, in-place header upgrading.
(upgrade_types): Rename to...
(upgrade_types_v1): ... this. Free ctf_dynbase, not ctf_base. No
longer track old and new headers separately. No longer allow for
header sizes explicitly: squeeze the headers out on upgrade (they
are preserved in fp->ctf_header). Set ctf_dynbase, ctf_base and
ctf_buf explicitly. Use ctf_free, not ctf_free_base.
(upgrade_types): New, also handle ctf_parmax updating.
(flip_header): Flip ctf_cuname.
(flip_types): Flip BUF explicitly rather than deriving BUF from
BASE.
(ctf_bufopen): Store the header in fp->ctf_header. Correct minimum
required alignment of objtoff and funcoff. No longer store it in
the ctf_buf unless that buf is derived unmodified from the input.
Set ctf_dynbase where ctf_base is dynamically allocated. Drop locals
that duplicate fields in ctf_file: move allocation of ctf_file
further up instead. Call upgrade_header as needed. Move
version-specific ctf_parmax initialization into upgrade_types. More
concise error handling.
(ctf_file_close): No longer test for null pointers before freeing.
Free ctf_dyncuname, ctf_dynbase, and ctf_header. Do not call
ctf_free_base.
(ctf_cuname): New.
(ctf_cuname_set): New.
* ctf-create.c (ctf_update): Populate ctf_cuname.
(ctf_gzwrite): Write out the header explicitly. Remove obsolescent
comment.
(ctf_write): Likewise.
(ctf_compress_write): Get the header from ctf_header, not ctf_base.
Fix the compression length: fp->ctf_size never counted the CTF
header. Simplify the compress call accordingly.
2019-07-07 00:36:21 +08:00
|
|
|
resid = sizeof (ctf_header_t);
|
|
|
|
buf = (unsigned char *) fp->ctf_header;
|
|
|
|
while (resid != 0)
|
|
|
|
{
|
|
|
|
if ((len = write (fd, buf, resid)) <= 0)
|
|
|
|
return (ctf_set_errno (fp, errno));
|
|
|
|
resid -= len;
|
|
|
|
buf += len;
|
|
|
|
}
|
|
|
|
|
|
|
|
resid = fp->ctf_size;
|
|
|
|
buf = fp->ctf_buf;
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
while (resid != 0)
|
|
|
|
{
|
libctf: allow the header to change between versions
libctf supports dynamic upgrading of the type table as file format
versions change, but before now has not supported changes to the CTF
header. Doing this is complicated by the baroque storage method used:
the CTF header is kept prepended to the rest of the CTF data, just as
when read from the file, and written out from there, and is
endian-flipped in place.
This makes accessing it needlessly hard and makes it almost impossible
to make the header larger if we add fields. The general storage
machinery around the malloced ctf pointer (the 'ctf_base') is also
overcomplicated: the pointer is sometimes malloced locally and sometimes
assigned from a parameter, so freeing it requires checking to see if
that parameter was used, needlessly coupling ctf_bufopen and
ctf_file_close together.
So split the header out into a new ctf_file_t.ctf_header, which is
written out explicitly: squeeze it out of the CTF buffer whenever we
reallocate it, and use ctf_file_t.ctf_buf to skip past the header when
we do not need to reallocate (when no upgrading or endian-flipping is
required). We now track whether the CTF base can be freed explicitly
via a new ctf_dynbase pointer which is non-NULL only when freeing is
possible.
With all this done, we can upgrade the header on the fly and add new
fields as desired, via a new upgrade_header function in ctf-open.
As with other forms of upgrading, libctf upgrades older headers
automatically to the latest supported version at open time.
For a first use of this field, we add a new string field cth_cuname, and
a corresponding setter/getter pair ctf_cuname_set and ctf_cuname: this
is used by debuggers to determine whether a CTF section's types relate
to a single compilation unit, or to all compilation units in the
program. (Types with ambiguous definitions in different CUs have only
one of these types placed in the top-level shared .ctf container: the
rest are placed in much smaller per-CU containers, which have the shared
container as their parent. Since CTF must be useful in the absence of
DWARF, we store the names of the relevant CUs ourselves, so the debugger
can look them up.)
v5: fix tabdamage.
include/
* ctf-api.h (ctf_cuname): New function.
(ctf_cuname_set): Likewise.
* ctf.h: Improve comment around upgrading, no longer
implying that v2 is the target of upgrades (it is v3 now).
(ctf_header_v2_t): New, old-format header for backward
compatibility.
(ctf_header_t): Add cth_cuname: this is the first of several
header changes in format v3.
libctf/
* ctf-impl.h (ctf_file_t): New fields ctf_header, ctf_dynbase,
ctf_cuname, ctf_dyncuname: ctf_base and ctf_buf are no longer const.
* ctf-open.c (ctf_set_base): Preserve the gap between ctf_buf and
ctf_base: do not assume that it is always sizeof (ctf_header_t).
Print out ctf_cuname: only print out ctf_parname if set.
(ctf_free_base): Removed, ctf_base is no longer freed: free
ctf_dynbase instead.
(ctf_set_version): Fix spacing.
(upgrade_header): New, in-place header upgrading.
(upgrade_types): Rename to...
(upgrade_types_v1): ... this. Free ctf_dynbase, not ctf_base. No
longer track old and new headers separately. No longer allow for
header sizes explicitly: squeeze the headers out on upgrade (they
are preserved in fp->ctf_header). Set ctf_dynbase, ctf_base and
ctf_buf explicitly. Use ctf_free, not ctf_free_base.
(upgrade_types): New, also handle ctf_parmax updating.
(flip_header): Flip ctf_cuname.
(flip_types): Flip BUF explicitly rather than deriving BUF from
BASE.
(ctf_bufopen): Store the header in fp->ctf_header. Correct minimum
required alignment of objtoff and funcoff. No longer store it in
the ctf_buf unless that buf is derived unmodified from the input.
Set ctf_dynbase where ctf_base is dynamically allocated. Drop locals
that duplicate fields in ctf_file: move allocation of ctf_file
further up instead. Call upgrade_header as needed. Move
version-specific ctf_parmax initialization into upgrade_types. More
concise error handling.
(ctf_file_close): No longer test for null pointers before freeing.
Free ctf_dyncuname, ctf_dynbase, and ctf_header. Do not call
ctf_free_base.
(ctf_cuname): New.
(ctf_cuname_set): New.
* ctf-create.c (ctf_update): Populate ctf_cuname.
(ctf_gzwrite): Write out the header explicitly. Remove obsolescent
comment.
(ctf_write): Likewise.
(ctf_compress_write): Get the header from ctf_header, not ctf_base.
Fix the compression length: fp->ctf_size never counted the CTF
header. Simplify the compress call accordingly.
2019-07-07 00:36:21 +08:00
|
|
|
if ((len = write (fd, buf, resid)) <= 0)
|
libctf: creation functions
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
2019-04-24 05:45:46 +08:00
|
|
|
return (ctf_set_errno (fp, errno));
|
|
|
|
resid -= len;
|
|
|
|
buf += len;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|