how to trace multiple threads on linux
--------------------------------------

prctl can trace a multi-threaded target program.  the
basic model is that prctl is ptrace-attached to every
individual target thread.

prctl requires mechanisms to detect thread creation,
breakpoint, and termination.  these are described
below.


the way we multiplex these detection mechanisms is by
calling waitpid with the -1 (any child process) pid
argument.

while we would like to treat threads identically,
normal processes and cloned processes are different
from the perspective of waitpid.  ordinarily,
waitpid(-1, x, 0) returns events on normal processes
but not cloned processes.  waitpid(-1, x, __WCLONE)
returns events on cloned processes.  waitpid(-1, x,
__WALL) returns events on either type of process.


creation
--------

the ptrace PTRACE_SETOPTIONS request has a flag,
PTRACE_O_TRACECLONE, for registering interest in new
threads cloned by the target thread.  setting this
option causes newly cloned threads to get a SIGSTOP
after it is created but before it starts to execute.

when the target thread PID clones a new thread,
waitpid(PID, &ST, __WALL) returns with

	     WSTOPSIG(st) == SIGTRAP && WSTOPEVENT(st) == PTRACE_EVENT_CLONE

then the ptrace PTRACE_GETEVENTMSG request on PID
returns the pid of the new thread.

the new thread must be waited upon for its SIGSTOP.
then the new thread must be continued in the usual way.

waitpid(-1, x, __WALL) may return for the SIGSTOP
of the newly cloned thread *before* its parent
stops for the PTRACE_EVENT_CLONE event.  thus
the debugger must be prepared to see SIGSTOP
events for processes it has never heard of.
when this happens, prctl remembers the event,
and later, when handling the PTRACE_EVENT_CLONE,
forgoes waiting for the new clone's SIGSTOP.


breakpoints
-----------

assuming threads share their address space,
an int3 breakpoint set in one thread
will cause any thread to SIGTRAP when executing
the instruction.

the SIGTRAP arrives as an event on the thread
that hit the breakpoint, and can be handled
as it would be for a single-thread program.

note, however, that other threads will continue to
execute.  this matters if the way the breakpoint
is handled is to reset and step over the trapped
instruction.


termination
-----------

waitpid returns notice of deatch of a thread the
same way it does for a single-threaded program.




linux makes it difficult for a process to
simultaneously wait for a change of state
to a set of traced processes and for a change
of state to a file descriptor.