Systems like mingw64 have pointers that can only be represented by 'long
long'. Consistently cast integers stored in pointers through uintptr_t
to cater for this.
libctf/
* ctf-create.c (ctf_dtd_insert): Add uintptr_t casts.
(ctf_dtd_delete): Likewise.
(ctf_dtd_lookup): Likewise.
(ctf_rollback): Likewise.
* ctf-hash.c (ctf_hash_lookup_type): Likewise.
* ctf-types.c (ctf_lookup_by_rawhash): Likewise.
This adds the core deduplicator that the ctf_link machinery calls
(possibly repeatedly) to link the CTF sections: it takes an array
of input ctf_file_t's and another array that indicates which entries in
the input array are parents of which other entries, and returns an array
of outputs. The first output is always the ctf_file_t on which
ctf_link/ctf_dedup/etc was called: the other outputs are child dicts
that have the first output as their parent.
include/
* ctf-api.h (CTF_LINK_SHARE_DUPLICATED): No longer unimplemented.
libctf/
* ctf-impl.h (ctf_type_id_key): New, the key in the
cd_id_to_file_t.
(ctf_dedup): New, core deduplicator state.
(ctf_file_t) <ctf_dedup>: New.
<ctf_dedup_atoms>: New.
<ctf_dedup_atoms_alloc>: New.
(ctf_hash_type_id_key): New prototype.
(ctf_hash_eq_type_id_key): Likewise.
(ctf_dedup_atoms_init): Likewise.
* ctf-hash.c (ctf_hash_eq_type_id_key): New.
(ctf_dedup_atoms_init): Likewise.
* ctf-create.c (ctf_serialize): Adjusted.
(ctf_add_encoded): No longer static.
(ctf_add_reftype): Likewise.
* ctf-open.c (ctf_file_close): Destroy the
ctf_dedup_atoms_alloc.
* ctf-dedup.c: New file.
* ctf-decls.h [!HAVE_DECL_STPCPY]: Add prototype.
* configure.ac: Check for stpcpy.
* Makefile.am: Add it.
* Makefile.in: Regenerate.
* config.h.in: Regenerate.
* configure: Regenerate.
The name was just annoyingly long and I kept misspelling it.
It's also a bad name: it's not a mapping the type might be *used* in a
type mapping, but it is itself a representation of a type (a ctf_file_t
/ ctf_id_t pair), not of a mapping at all.
libctf/
* ctf-impl.h (ctf_link_type_mapping_key): Rename to...
(ctf_link_type_key): ... this, adjusting member prefixes to
match.
(ctf_hash_type_mapping_key): Rename to...
(ctf_hash_type_key): ... this.
(ctf_hash_eq_type_mapping_key): Rename to...
(ctf_hash_eq_type_key): ... this.
* ctf-hash.c (ctf_hash_type_mapping_key): Rename to...
(ctf_hash_type_key): ... this, and adjust for member name
changes.
(ctf_hash_eq_type_mapping_key): Rename to...
(ctf_hash_eq_type_key): ... this, and adjust for member name
changes.
* ctf-link.c (ctf_add_type_mapping): Adjust. Note the lack of
need for out-of-memory checking in this code.
(ctf_type_mapping): Adjust.
This lets you iterate over dynhashes and dynsets using the _next API.
dynhashes can be iterated over in sorted order, which works by
populating an array of key/value pairs using ctf_dynhash_next itself,
then sorting it with qsort.
Convenience inline functions named ctf_dyn{hash,set}_cnext are also
provided that take (-> return) const keys and values.
libctf/
* ctf-impl.h (ctf_next_hkv_t): New, kv-pairs passed to
sorting functions.
(ctf_next_t) <u.ctn_sorted_hkv>: New, sorted kv-pairs for
ctf_dynhash_next_sorted.
<cu.ctn_h>: New, pointer to the dynhash under iteration.
<cu.ctn_s>: New, pointer to the dynset under iteration.
(ctf_hash_sort_f): Sorting function passed to...
(ctf_dynhash_next_sorted): ... this new function.
(ctf_dynhash_next): New.
(ctf_dynset_next): New.
* ctf-inlines.h (ctf_dynhash_cnext_sorted): New.
(ctf_dynhash_cnext): New.
(ctf_dynset_cnext): New.
* ctf-hash.c (ctf_dynhash_next_sorted): New.
(ctf_dynhash_next): New.
(ctf_dynset_next): New.
* ctf-util.c (ctf_next_destroy): Free the u.ctn_sorted_hkv if
needed.
(ctf_next_copy): Alloc-and-copy the u.ctn_sorted_hkv if needed.
There are many places in the deduplicator which use hashtables as tiny
sets: keys with no value (and usually, but not always, no freeing
function) often with only one or a few members. For each of these, even
after the last change to not store the freeing functions, we are storing
a little malloced block for each item just to track the key/value pair,
and a little malloced block for the hash table itself just to track the
freeing function because we can't use libiberty hashtab's freeing
function because we are using that to free the little malloced per-item
block.
If we only have a key, we don't need any of that: we can ditch the
per-malloced block because we don't have a value, and we can ditch the
per-hashtab structure because we don't need to independently track the
freeing functions since libiberty hashtab is doing it for us. That
means we don't need an owner field in the (now nonexistent) item block
either.
Roughly speaking, this datatype saves about 25% in time and 20% in peak
memory usage for normal links, even fairly big ones. So this might seem
redundant, but it's really worth it.
Instead of a _lookup function, a dynset has two distinct functions:
ctf_dynset_exists, which returns true or false and an optional pointer
to the set member, and ctf_dynhash_lookup_any, which is used if all
members of the set are expected to be equivalent and we just want *any*
member and we don't care which one.
There is no iterator in this set of functions, not because we don't
iterate over dynset members -- we do, a lot -- but because the iterator
here is a member of an entirely new family of much more convenient
iteration functions, introduced in the next commit.
libctf/
* ctf-hash.c (ctf_dynset_eq_string): New.
(ctf_dynset_create): New.
(DYNSET_EMPTY_ENTRY_REPLACEMENT): New.
(DYNSET_DELETED_ENTRY_REPLACEMENT): New.
(key_to_internal): New.
(internal_to_key): New.
(ctf_dynset_insert): New.
(ctf_dynset_remove): New.
(ctf_dynset_destroy): New.
(ctf_dynset_lookup): New.
(ctf_dynset_exists): New.
(ctf_dynset_lookup_any): New.
(ctf_hash_insert_type): Coding style.
(ctf_hash_define_type): Likewise.
* ctf-impl.h (ctf_dynset_t): New.
(ctf_dynset_eq_string): New.
(ctf_dynset_create): New.
(ctf_dynset_insert): New.
(ctf_dynset_remove): New.
(ctf_dynset_destroy): New.
(ctf_dynset_lookup): New.
(ctf_dynset_exists): New.
(ctf_dynset_lookup_any): New.
* ctf-inlines.h (ctf_dynset_cinsert): New.
The libctf dynhash hashtab abstraction supports per-hashtab arbitrary
key/item freeing functions -- but it also has a constant slot type that
holds both key and value requested by the user, so it needs to use its
own freeing function to free that -- and it has nowhere to store the
freeing functions the caller requested.
So it copies them into every hash item, bloating every slot, even though
all items in a given hash table must have the same key and value freeing
functions.
So point back to the owner using a back-pointer, but don't even spend
space in the item or the hashtab allocating those freeing functions
unless necessary: if none are needed, we can simply arrange to not pass
in ctf_dynhash_item_free as a del_f to hashtab_create_alloc, and none of
those fields will ever be accessed.
The only downside is that this makes the code sensitive to the order of
fields in the ctf_helem_t and ctf_hashtab_t: but the deduplicator
allocates so many hash tables that doing this alone cuts memory usage
during deduplication by about 10%. (libiberty hashtab itself has a lot
of per-hashtab bloat: in the future we might trim that down, or make a
trimmer version.)
libctf/
* ctf-hash.c (ctf_helem_t) <key_free>: Remove.
<value_free>: Likewise.
<owner>: New.
(ctf_dynhash_item_free): Indirect through the owner.
(ctf_dynhash_create): Only pass in ctf_dynhash_item_free and
allocate space for the key_free and value_free fields fields
if necessary.
(ctf_hashtab_insert): Likewise. Fix OOM errno value.
(ctf_dynhash_insert): Only access ctf_hashtab's key_free and
value_free if they will exist. Set the slot's owner, but only
if it exists.
(ctf_dynhash_remove): Adjust.
Right now, if you insert a key/value pair into a dynhash, the old slot's
key is freed and the new one always assigned. This seemed sane to me
when I wrote it, but I got it wrong time and time again. It's much
less confusing to free the key passed in: if a key-freeing function
was passed, you are asserting that the dynhash owns the key in any
case, so if you pass in a key it is always buggy to assume it sticks
around. Freeing the old key means that you can't even safely look up a
key from out of a dynhash and hold on to it, because some other matching
key might force it to be freed at any time.
In the new model, you can always get a key out of a dynhash with
ctf_dynhash_lookup_kv and hang on to it until the kv-pair is actually
deleted from the dynhash. In the old model the pointer to the key might
be freed at any time if a matching key was inserted.
libctf/
* ctf-hash.c (ctf_hashtab_insert): Free the key passed in if
there is a key-freeing function and the key already exists.
Future commits will use these.
ctf_dynhash_elements: count elements in a dynhash
ctf_dynhash_lookup_kv: look up and return pointers to the original key
and value in a dynhash (the only way of getting
a reference to the original key)
ctf_dynhash_iter_find: iterate until an item is found, then return its
key
ctf_dynhash_cinsert: insert a const key / value into a dynhash (a thim
wrapper in a new header dedicated to inline
functions).
As with the rest of ctf_dynhash, this is not public API. No impact
on existing callers is expected.
libctf/
* ctf-inlines.h: New file.
* ctf-impl.h: Include it.
(ctf_hash_iter_find_f): New typedef.
(ctf_dynhash_elements): New.
(ctf_dynhash_lookup_kv): New.
(ctf_dynhash_iter_find): New.
* ctf-hash.c (ctf_dynhash_lookup_kv): New.
(ctf_traverse_find_cb_arg_t): New.
(ctf_hashtab_traverse_find): New.
(ctf_dynhash_iter_find): New.
(ctf_dynhash_elements): New.
When a ctf_dynhash_insert() finds a slot already existing, it should
call the key and value free functions on the existing key and value and
move the passed-in key into place, so that the lifetime rules for hash
keys are always the same no matter whether the key existed or not but
neither are the keys or values leaked.
New in v3.
v5: fix tabdamage.
libctf/
* ctf-hash.c (ctf_hashtab_insert): Pass in the key and value
freeing functions: if set, free the key and value if the slot
already exists. Always reassign the key.
(ctf_dynhash_insert): Adjust call appropriately.
(ctf_hash_insert_type): Likewise.
This lets you call ctf_type_mapping (dest_fp, src_fp, src_type_id)
and get told what type ID the corresponding type has in the target
ctf_file_t. This works even if it was added by a recursive call, and
because it is stored in the target ctf_file_t it works even if we
had to add one type to multiple ctf_file_t's as part of conflicting
type handling.
We empty out this mapping after every archive is linked: because it maps
input to output fps, and we only visit each input fp once, its contents
are rendered entirely useless every time the source fp changes.
v3: add several missing mapping additions. Add ctf_dynhash_empty, and
empty after every input archive.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t): New field ctf_link_type_mapping.
(struct ctf_link_type_mapping_key): New.
(ctf_hash_type_mapping_key): Likewise.
(ctf_hash_eq_type_mapping_key): Likewise.
(ctf_add_type_mapping): Likewise.
(ctf_type_mapping): Likewise.
(ctf_dynhash_empty): Likewise.
* ctf-open.c (ctf_file_close): Update accordingly.
* ctf-create.c (ctf_update): Likewise.
(ctf_add_type): Populate the mapping.
* ctf-hash.c (ctf_hash_type_mapping_key): Hash a type mapping key.
(ctf_hash_eq_type_mapping_key): Check the key for equality.
(ctf_dynhash_insert): Fix comment typo.
(ctf_dynhash_empty): New.
* ctf-link.c (ctf_add_type_mapping): New.
(ctf_type_mapping): Likewise.
(empty_link_type_mapping): New.
(ctf_link_one_input_archive): Call it.
The CTF file format has always supported "external strtabs", which
internally are strtab offsets with their MSB on: such refs
get their strings from the strtab passed in at CTF file open time:
this is usually intended to be the ELF strtab, and that's what this
implementation is meant to support, though in theory the external
strtab could come from anywhere.
This commit adds support for these external strings in the ctf-string.c
strtab tracking layer. It's quite easy: we just add a field csa_offset
to the atoms table that tracks all strings: this field tracks the offset
of the string in the ELF strtab (with its MSB already on, courtesy of a
new macro CTF_SET_STID), and adds a new function that sets the
csa_offset to the specified offset (plus MSB). Then we just need to
avoid writing out strings to the internal strtab if they have csa_offset
set, and note that the internal strtab is shorter than it might
otherwise be.
(We could in theory save a little more time here by eschewing sorting
such strings, since we never actually write the strings out anywhere,
but that would mean storing them separately and it's just not worth the
complexity cost until profiling shows it's worth doing.)
We also have to go through a bit of extra effort at variable-sorting
time. This was previously using direct references to the internal
strtab: it couldn't use ctf_strptr or ctf_strraw because the new strtab
is not yet ready to put in its usual field (in a ctf_file_t that hasn't
even been allocated yet at this stage): but now we're using the external
strtab, this will no longer do because it'll be looking things up in the
wrong strtab, with disastrous results. Instead, pass the new internal
strtab in to a new ctf_strraw_explicit function which is just like
ctf_strraw except you can specify a ne winternal strtab to use.
But even now that it is using a new internal strtab, this is not quite
enough: it can't look up strings in the external strtab because ld
hasn't written it out yet, and when it does will write it straight to
disk. Instead, when we write the internal strtab, note all the offset
-> string mappings that we have noted belong in the *external* strtab to
a new "synthetic external strtab" dynhash, ctf_syn_ext_strtab, and look
in there at ctf_strraw time if it is set. This uses minimal extra
memory (because only strings in the external strtab that we actually use
are stored, and even those come straight out of the atoms table), but
let both variable sorting and name interning when ctf_bufopen is next
called work fine. (This also means that we don't need to filter out
spurious ECTF_STRTAB warnings from ctf_bufopen but can pass them back to
the caller, once we wrap ctf_bufopen so that we have a new internal
variant of ctf_bufopen etc that we can pass the synthetic external
strtab to. That error has been filtered out since the days of Solaris
libctf, which didn't try to handle the problem of getting external
strtabs right at construction time at all.)
v3: add the synthetic strtab and all associated machinery.
v5: fix tabdamage.
include/
* ctf.h (CTF_SET_STID): New.
libctf/
* ctf-impl.h (ctf_str_atom_t) <csa_offset>: New field.
(ctf_file_t) <ctf_syn_ext_strtab>: Likewise.
(ctf_str_add_ref): Name the last arg.
(ctf_str_add_external) New.
(ctf_str_add_strraw_explicit): Likewise.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
* ctf-string.c (ctf_strraw_explicit): Split from...
(ctf_strraw): ... here, with new support for ctf_syn_ext_strtab.
(ctf_str_add_ref_internal): Return the atom, not the
string.
(ctf_str_add): Adjust accordingly.
(ctf_str_add_ref): Likewise. Move up in the file.
(ctf_str_add_external): New: update the csa_offset.
(ctf_str_count_strtab): Only account for strings with no csa_offset
in the internal strtab length.
(ctf_str_write_strtab): If the csa_offset is set, update the
string's refs without writing the string out, and update the
ctf_syn_ext_strtab. Make OOM handling less ugly.
* ctf-create.c (struct ctf_sort_var_arg_cb): New.
(ctf_update): Handle failure to populate the strtab. Pass in the
new ctf_sort_var arg. Adjust for ctf_syn_ext_strtab addition.
Call ctf_simple_open_internal, not ctf_simple_open.
(ctf_sort_var): Call ctf_strraw_explicit rather than looking up
strings by hand.
* ctf-hash.c (ctf_hash_insert_type): Likewise (but using
ctf_strraw). Adjust to diagnose ECTF_STRTAB nonetheless.
* ctf-open.c (init_types): No longer filter out ECTF_STRTAB.
(ctf_file_close): Destroy the ctf_syn_ext_strtab.
(ctf_simple_open): Rename to, and reimplement as a wrapper around...
(ctf_simple_open_internal): ... this new function, which calls
ctf_bufopen_internal.
(ctf_bufopen): Rename to, and reimplement as a wrapper around...
(ctf_bufopen_internal): ... this new function, which sets
ctf_syn_ext_strtab.
There are two, ctf_dynhash_iter and ctf_dynhash_iter_remove: the latter
lets you return a nonzero value to remove the element being iterated
over.
Used in the next commit.
libctf/
* ctf-impl.h (ctf_hash_iter_f): New.
(ctf_dynhash_iter): New declaration.
(ctf_dynhash_iter_remove): New declaration.
* ctf-hash.c (ctf_dynhash_iter): Define.
(ctf_dynhash_iter_remove): Likewise.
(ctf_hashtab_traverse): New.
(ctf_hashtab_traverse_remove): Likewise.
(struct ctf_traverse_cb_arg): Likewise.
(struct ctf_traverse_remove_cb_arg): Likewise.
We must call htab_remove_elt with an element (in this case, a mocked-up
one with only the key populated, since no reasonable hash function will
need the other fields), not with the key alone.
libctf/
* ctf-hash.c (ctf_dynhash_remove): Call with a mocked-up element.
libctf maintains two distinct hash ADTs, one (ctf_dynhash) for wrapping
dynamically-generated unknown-sized hashes during CTF file construction,
one (ctf_hash) for wrapping unchanging hashes whose size is known at
creation time for reading CTF files that were previously created.
In the binutils implementation, these are both fairly thin wrappers
around libiberty hashtab.
Unusually, this code is not kept synchronized with libdtrace-ctf,
due to its dependence on libiberty hashtab.
libctf/
* ctf-hash.c: New file.
* ctf-impl.h: New declarations.
