This example illustrates the use of PDT, hand-instrumentation (for threads), 
MPI library instrumentation and TAU system call wrapper library instrumentation.
To use this, you must configure TAU using:
% configure -mpiinc=<dir> -mpilib=<dir> -pdt=<dir> -pthread

The application spawns a thread. The thread calls work() which calls sleep, getpid and write calls. We want to track the time spent in system calls getpid, sleep and write (invoked by work() in processing.cpp). The Makefile is organized
such that TAU is enabled or disabled by uncommenting or commenting the USE_TAU variable in the Makefile respectively. 

Note: since the threaded function cannot be instrumented using
PDT, we put it in thr.c and have a special rule for compiling this file
(not using PDT). The default .cpp.o rule compiles all files (app.cpp,
processing.cpp) using PDT.

Run the example using:
        % mpirun -np 2 app
        and you'll see that performance data on thread 1 in each context
has getpid, sleep and write routines. You'll also see that MPI calls are
present on thread 1 (rank 0 sends an int to rank 1). This can help you build 
an appropriate syscall wrapper library to instrument those functions whose 
source code is not available for instrumentation. 
 

[sameer@neutron mixedmode]$ ls
Makefile  TauWrapper.cpp  TauWrapper.h	app.cpp  processing.cpp  thr.cpp
[sameer@neutron mixedmode]$ cat ../../.last_config 
-pdt=/scratch1/sameer/pdt/pdtoolkit-2.0 -mpiinc=/usr/local/packages/mpich/include -mpilib=/usr/local/packages/mpich/lib -pthread
[sameer@neutron mixedmode]$ head -15 Makefile
#****************************************************************************
#*			TAU Portable Profiling Package			   **
#*			http://www.cs.uoregon.edu/research/tau	           **
#****************************************************************************
#*    Copyright 2001  						   	   **
#*    Department of Computer and Information Science, University of Oregon **
#*    Advanced Computing Laboratory, Los Alamos National Laboratory        **
#****************************************************************************
 
TAUROOTDIR	= ../..

include $(TAUROOTDIR)/include/Makefile

USE_TAU = 1
# Comment above line to disable TAU 
[sameer@neutron mixedmode]$ make
make: *** Warning: File `../../include/Makefile' has modification time in the future (2001-12-10 21:17:14 > 2001-12-10 15:24:32)
/scratch1/sameer/pdt/pdtoolkit-2.0/i386_linux/bin/cxxparse app.cpp -I/home/users/sameer/tau2/include -DPROFILING_ON                   -DTAU_STDCXXLIB          -DNO_RTTI               -DTAU_GNU                -DPTHREADS  	                  -DTAU_MPI      -I/usr/local/packages/mpich/include; ../../i386_linux/bin/tau_instrumentor app.pdb app.cpp -o app.inst.cpp ; g++    -I/home/users/sameer/tau2/include -DPROFILING_ON                   -DTAU_STDCXXLIB          -DNO_RTTI               -DTAU_GNU                -DPTHREADS  	                  -DTAU_MPI      -I/usr/local/packages/mpich/include -c app.inst.cpp -o app.o ; rm -f app.pdb ;      
	edgcpfe app.cpp --> /tmp/13928.il
	taucpdisp -t /tmp/13928.il --> app.pdb
g++    -I/home/users/sameer/tau2/include -DPROFILING_ON                   -DTAU_STDCXXLIB          -DNO_RTTI               -DTAU_GNU                -DPTHREADS  	                  -DTAU_MPI      -I/usr/local/packages/mpich/include -c thr.cpp -o thr.o
/scratch1/sameer/pdt/pdtoolkit-2.0/i386_linux/bin/cxxparse processing.cpp -I/home/users/sameer/tau2/include -DPROFILING_ON                   -DTAU_STDCXXLIB          -DNO_RTTI               -DTAU_GNU                -DPTHREADS  	                  -DTAU_MPI      -I/usr/local/packages/mpich/include; ../../i386_linux/bin/tau_instrumentor processing.pdb processing.cpp -o processing.inst.cpp ; g++    -I/home/users/sameer/tau2/include -DPROFILING_ON                   -DTAU_STDCXXLIB          -DNO_RTTI               -DTAU_GNU                -DPTHREADS  	                  -DTAU_MPI      -I/usr/local/packages/mpich/include -c processing.inst.cpp -o processing.o ; rm -f processing.pdb ;      
	edgcpfe processing.cpp --> /tmp/13943.il
	taucpdisp -t /tmp/13943.il --> processing.pdb
g++    -I/home/users/sameer/tau2/include -DPROFILING_ON                   -DTAU_STDCXXLIB          -DNO_RTTI               -DTAU_GNU                -DPTHREADS  	                  -DTAU_MPI      -I/usr/local/packages/mpich/include -c TauWrapper.cpp -o TauWrapper.o
ar rcv libTauWrapper.a TauWrapper.o
a - TauWrapper.o
g++     app.o  thr.o processing.o  libTauWrapper.a  -o app -L/home/users/sameer/tau2/i386_linux/lib -lTauMpi-pthread -L/usr/local/packages/mpich/lib -lpmpich -lmpich  -L/home/users/sameer/tau2/i386_linux/lib -ltau-pthread  -lpthread	         -lm
make: *** Warning:  Clock skew detected.  Your build may be incomplete.
[sameer@neutron mixedmode]$ ls
Makefile	TauWrapper.o  app.inst.cpp     processing.cpp	    thr.cpp
TauWrapper.cpp	app	      app.o	       processing.inst.cpp  thr.o
TauWrapper.h	app.cpp       libTauWrapper.a  processing.o
[sameer@neutron mixedmode]$ mpirun -np 2 app
After Initialization - my rank is 0 out of 2 procs
Inside work (called from threaded_func): rank 0, pid = 14057
Inside work (called from threaded_func): rank 1, pid = 14056
After Initialization - my rank is 1 out of 2 procs
rank = 1:Received data = 5767, tag = 34, source = 0
[sameer@neutron mixedmode]$ ls
Makefile	app	      libTauWrapper.a	   profile.0.0.0  thr.cpp
TauWrapper.cpp	app.cpp       processing.cpp	   profile.0.0.1  thr.o
TauWrapper.h	app.inst.cpp  processing.inst.cpp  profile.1.0.0
TauWrapper.o	app.o	      processing.o	   profile.1.0.1
[sameer@neutron mixedmode]$ pprof
Reading Profile files in profile.*

NODE 0;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name
              msec   total msec                          usec/call 
---------------------------------------------------------------------------------------
100.0            1        2,202           1           5    2202686 int main(int, char **)  
 91.4        2,011        2,012           1           2    2012498 MPI_Finalize()  
  8.6          188          188           1          35     188501 MPI_Init()  
  0.0        0.648         0.65           1           1        650 MPI_Sendrecv()  
  0.0        0.291        0.291           1           0        291 MPI_Send()  
  0.0        0.105        0.105           1           0        105 MPI_Keyval_create()  
  0.0        0.044        0.044           5           0          9 MPI_Type_struct()  
  0.0        0.025        0.025           8           0          3 MPI_Attr_put()  
  0.0        0.025        0.025           6           0          4 MPI_Type_contiguous()  
  0.0        0.021        0.021          11           0          2 MPI_Type_commit()  
  0.0        0.012        0.012           4           0          3 MPI_Errhandler_set()  
  0.0        0.007        0.007           1           0          7 MPI_Keyval_free()  
  0.0        0.002        0.002           1           0          2 MPI_Comm_rank()  
  0.0        0.002        0.002           1           0          2 MPI_Comm_size()  
  0.0        0.002        0.002           1           0          2 MPI_Type_size()  

NODE 0;CONTEXT 0;THREAD 1:
---------------------------------------------------------------------------------------
%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name
              msec   total msec                          usec/call 
---------------------------------------------------------------------------------------
100.0        0.061        2,005           1           1    2005330 threaded_func() int ()
100.0        0.164        2,005           1           3    2005269 int work(int)  
100.0        2,005        2,005           1           0    2005039 sleep() int (int)
  0.0        0.063        0.063           1           0         63 write() int (int, char *, int)
  0.0        0.003        0.003           1           0          3 getpid() pid_t ()

NODE 1;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name
              msec   total msec                          usec/call 
---------------------------------------------------------------------------------------
100.0        2,006        2,037           1           5    2037470 int main(int, char **)  
  1.4           28           28           1          35      28415 MPI_Init()  
  0.1            1            1           1           2       1873 MPI_Finalize()  
  0.0        0.507        0.507           1           0        507 MPI_Recv()  
  0.0        0.471        0.475           1           1        475 MPI_Sendrecv()  
  0.0        0.148        0.148           1           0        148 MPI_Keyval_create()  
  0.0        0.043        0.043           5           0          9 MPI_Type_struct()  
  0.0        0.028        0.028           6           0          5 MPI_Type_contiguous()  
  0.0        0.026        0.026          11           0          2 MPI_Type_commit()  
  0.0        0.024        0.024           8           0          3 MPI_Attr_put()  
  0.0        0.014        0.014           4           0          4 MPI_Errhandler_set()  
  0.0        0.006        0.006           1           0          6 MPI_Keyval_free()  
  0.0        0.004        0.004           1           0          4 MPI_Type_size()  
  0.0        0.003        0.003           1           0          3 MPI_Comm_size()  
  0.0        0.002        0.002           1           0          2 MPI_Comm_rank()  

NODE 1;CONTEXT 0;THREAD 1:
---------------------------------------------------------------------------------------
%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name
              msec   total msec                          usec/call 
---------------------------------------------------------------------------------------
100.0         0.09        2,005           1           1    2005272 threaded_func() int ()
100.0        0.215        2,005           1           3    2005182 int work(int)  
100.0        2,004        2,004           1           0    2004914 sleep() int (int)
  0.0         0.05         0.05           1           0         50 write() int (int, char *, int)
  0.0        0.003        0.003           1           0          3 getpid() pid_t ()

FUNCTION SUMMARY (total):
---------------------------------------------------------------------------------------
%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name
              msec   total msec                          usec/call 
---------------------------------------------------------------------------------------
 51.4        2,008        4,240           2          10    2120078 int main(int, char **)  
 48.6        0.151        4,010           2           2    2005301 threaded_func() int ()
 48.6        0.379        4,010           2           6    2005226 int work(int)  
 48.6        4,009        4,009           2           0    2004976 sleep() int (int)
 24.4        2,013        2,014           2           4    1007186 MPI_Finalize()  
  2.6          216          216           2          70     108458 MPI_Init()  
  0.0            1            1           2           2        562 MPI_Sendrecv()  
  0.0        0.507        0.507           1           0        507 MPI_Recv()  
  0.0        0.291        0.291           1           0        291 MPI_Send()  
  0.0        0.253        0.253           2           0        126 MPI_Keyval_create()  
  0.0        0.113        0.113           2           0         56 write() int (int, char *, int)
  0.0        0.087        0.087          10           0          9 MPI_Type_struct()  
  0.0        0.053        0.053          12           0          4 MPI_Type_contiguous()  
  0.0        0.049        0.049          16           0          3 MPI_Attr_put()  
  0.0        0.047        0.047          22           0          2 MPI_Type_commit()  
  0.0        0.026        0.026           8           0          3 MPI_Errhandler_set()  
  0.0        0.013        0.013           2           0          6 MPI_Keyval_free()  
  0.0        0.006        0.006           2           0          3 MPI_Type_size()  
  0.0        0.006        0.006           2           0          3 getpid() pid_t ()
  0.0        0.005        0.005           2           0          2 MPI_Comm_size()  
  0.0        0.004        0.004           2           0          2 MPI_Comm_rank()  

FUNCTION SUMMARY (mean):
---------------------------------------------------------------------------------------
%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name
              msec   total msec                          usec/call 
---------------------------------------------------------------------------------------
 51.4          502        1,060         0.5         2.5    2120078 int main(int, char **)  
 48.6       0.0377        1,002         0.5         0.5    2005301 threaded_func() int ()
 48.6       0.0948        1,002         0.5         1.5    2005226 int work(int)  
 48.6        1,002        1,002         0.5           0    2004976 sleep() int (int)
 24.4          503          503         0.5           1    1007186 MPI_Finalize()  
  2.6           54           54         0.5        17.5     108458 MPI_Init()  
  0.0         0.28        0.281         0.5         0.5        562 MPI_Sendrecv()  
  0.0        0.127        0.127        0.25           0        507 MPI_Recv()  
  0.0       0.0727       0.0727        0.25           0        291 MPI_Send()  
  0.0       0.0633       0.0633         0.5           0        126 MPI_Keyval_create()  
  0.0       0.0283       0.0283         0.5           0         56 write() int (int, char *, int)
  0.0       0.0217       0.0217         2.5           0          9 MPI_Type_struct()  
  0.0       0.0132       0.0132           3           0          4 MPI_Type_contiguous()  
  0.0       0.0123       0.0123           4           0          3 MPI_Attr_put()  
  0.0       0.0118       0.0118         5.5           0          2 MPI_Type_commit()  
  0.0       0.0065       0.0065           2           0          3 MPI_Errhandler_set()  
  0.0      0.00325      0.00325         0.5           0          6 MPI_Keyval_free()  
  0.0       0.0015       0.0015         0.5           0          3 MPI_Type_size()  
  0.0       0.0015       0.0015         0.5           0          3 getpid() pid_t ()
  0.0      0.00125      0.00125         0.5           0          2 MPI_Comm_size()  
  0.0        0.001        0.001         0.5           0          2 MPI_Comm_rank()  
[sameer@neutron mixedmode]$ echo "Commenting USE_TAU variable in Makefile to disable TAU instrumentation"
Commenting USE_TAU variable in Makefile to disable TAU instrumentation
[sameer@neutron mixedmode]$ head -15 Makefile
#****************************************************************************
#*			TAU Portable Profiling Package			   **
#*			http://www.cs.uoregon.edu/research/tau	           **
#****************************************************************************
#*    Copyright 2001  						   	   **
#*    Department of Computer and Information Science, University of Oregon **
#*    Advanced Computing Laboratory, Los Alamos National Laboratory        **
#****************************************************************************
 
TAUROOTDIR	= ../..

include $(TAUROOTDIR)/include/Makefile

# USE_TAU = 1
# Comment above line to disable TAU 
[sameer@neutron mixedmode]$ make clean
make: *** Warning: File `../../include/Makefile' has modification time in the future (2001-12-10 21:17:14 > 2001-12-10 15:26:03)
/bin/rm -f app.o  thr.o processing.o  app app.inst.cpp  thr.inst.cpp processing.inst.cpp  \
TauWrapper.o libTauWrapper.a  profile.* tautrace.* events.*
make: *** Warning:  Clock skew detected.  Your build may be incomplete.
[sameer@neutron mixedmode]$ make
make: *** Warning: File `../../include/Makefile' has modification time in the future (2001-12-10 21:17:14 > 2001-12-10 15:26:07)
g++    -I/home/users/sameer/tau2/include  -I/usr/local/packages/mpich/include -c app.cpp -o app.o ;
g++    -I/home/users/sameer/tau2/include  -I/usr/local/packages/mpich/include -c thr.cpp -o thr.o
g++    -I/home/users/sameer/tau2/include  -I/usr/local/packages/mpich/include -c processing.cpp -o processing.o ;
g++     app.o  thr.o processing.o   -o app -L/usr/local/packages/mpich/lib -lmpich  -lpthread	         -lm
make: *** Warning:  Clock skew detected.  Your build may be incomplete.
[sameer@neutron mixedmode]$ ls
Makefile	TauWrapper.h  app.cpp  processing.cpp  thr.cpp
TauWrapper.cpp	app	      app.o    processing.o    thr.o
[sameer@neutron mixedmode]$ mpirun -np 2 app
After Initialization - my rank is 0 out of 2 procs
Inside work (called from threaded_func): rank 0, pid = 14184
Inside work (called from threaded_func): rank 1, pid = 14183
After Initialization - my rank is 1 out of 2 procs
rank = 1:Received data = 5767, tag = 34, source = 0
[sameer@neutron mixedmode]$ ls
Makefile	TauWrapper.h  app.cpp  processing.cpp  thr.cpp
TauWrapper.cpp	app	      app.o    processing.o    thr.o
